Importance weighting adjusts token contributions according to informativeness. In BERTScore, inverse document frequency (IDF) reduces the influence of common words and increases the influence of distinctive words. section 3.2.4
Apply importance weighting during BERTScore calculation by:
- Computing each reference token’s best semantic match in the candidate using embedding similarity. section 3.2.4
- Multiplying each maximum similarity by that token’s IDF. section 3.2.4
- Summing the weighted similarities and dividing by the sum of the IDFs. section 3.2.4
Common versus distinctive words
Matching “the” contributes relatively little because common words receive less importance, while matching “bureaucracy” contributes more. One worked calculation gives “the” a maximum similarity of 0.713 and an IDF of 1.27, producing a weighted similarity of 0.906; the resulting importance-weighted recall is 0.753. section 3.2.4
Figure 3.7 Computing BERTScore. Given the reference and candidate, we compute embeddings and pairwise cosine similarity. The heatmap highlights the greedy matching, where each reference token is matched to its best candidate token. Optional importance weighting emphasizes less common words over common ones. And we will scale the score to be more evenly distributed between 0 and 1.
Use the BERTScore diagram to trace embeddings, pairwise cosine similarity, greedy matching, and optional IDF emphasis in one workflow. section 3.2.4
