liveBook · Manning

Importance weighting

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Importance weighting adjusts individual contributions to a score according to their relative informativeness.

Importance weighting adjusts token contributions according to informativeness. In BERTScore, inverse document frequency (IDF) reduces the influence of common words and increases the influence of distinctive words. section 3.2.4

Apply importance weighting during BERTScore calculation by:

  • Computing each reference token’s best semantic match in the candidate using embedding similarity. section 3.2.4
  • Multiplying each maximum similarity by that token’s IDF. section 3.2.4
  • Summing the weighted similarities and dividing by the sum of the IDFs. section 3.2.4

Common versus distinctive words

Matching “the” contributes relatively little because common words receive less importance, while matching “bureaucracy” contributes more. One worked calculation gives “the” a maximum similarity of 0.713 and an IDF of 1.27, producing a weighted similarity of 0.906; the resulting importance-weighted recall is 0.753. section 3.2.4

Figure 3.7 Computing BERTScore. Given the reference and candidate, we compute embeddings and pairwise cosine similarity. The heatmap highlights the greedy matching, where each reference token is matched to its best candidate token. Optional importance weighting emphasizes less common words over common ones. And we will scale the score to be more evenly distributed between 0 and 1.

Use the BERTScore diagram to trace embeddings, pairwise cosine similarity, greedy matching, and optional IDF emphasis in one workflow. section 3.2.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines IDF weighting and works through weighted similarity and normalization, without developing the broader metric beyond its matching workflow.
Examples
Uses “the” and “bureaucracy” to contrast common and distinctive words, then calculates an importance-weighted BERTScore recall.
What it assumes
Assumes token embeddings, cosine similarity, and BERTScore matching while explaining IDF’s role through concrete calculations.
What you can do after
Apply it via BERTScore to emphasize distinctive semantic content when comparing candidate and reference sentences.
How it connects
Connects BERTScore with inverse document frequency, embeddings, cosine similarity, greedy matching, precision, and recall.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.