BERTScore is a reference-based metric for comparing the meaning of generated text with a reference. Unlike lexical metrics such as BLEU and ROUGE, it uses contextual language-model embeddings to recognize similar meanings expressed in different words. section 3.0 section 3.2
Recognizing a paraphrase
Against the reference “people like foreign cars,” the candidate “consumers prefer imported cars” conveys a similar meaning despite using different words. A word-overlap metric can penalize this valid paraphrase; BERTScore compares learned semantic representations instead. section 3.0
BERTScore uses a contextual embedding for each token: the representation depends on the surrounding text. For example, “bank” can have a different representation near “money” than near “river.” BERTScore is a metric, not the BERT model itself; it can use other underlying encoders, including RoBERTa. section 3.2.2
To compare a candidate with a reference:
- Encode both texts with a pretrained model to obtain contextual token embeddings. section 3.2.4
- Calculate cosine similarity between every candidate token and every reference token. section 3.2.4
- For precision, find each candidate token’s best reference match and average the similarities. For recall, reverse the direction: find each reference token’s best candidate match and average those similarities. section 3.2.4
- Optionally weight matches by inverse document frequency (IDF), giving distinctive words more influence than common words. Baseline rescaling can also calibrate scores. section 3.2.4 section 3.2.5
Figure 3.7 Computing BERTScore. Given the reference and candidate, we compute embeddings and pairwise cosine similarity. The heatmap highlights the greedy matching, where each reference token is matched to its best candidate token. Optional importance weighting emphasizes less common words over common ones. And we will scale the score to be more evenly distributed between 0 and 1.
The token-similarity heatmap illustrates greedy matching between a candidate and a reference. section 3.2.4
Use BERTScore when valid paraphrases matter, such as in summarization or translation, and a suitable reference is available. It is more computationally expensive and less interpretable than lexical overlap metrics. Exact-match metrics remain useful when a changed word can make an output invalid, as in SQL extraction. section 2.3.6 section 3.1.1 section 3.4.1
A high similarity score is not a guarantee of factual accuracy or overall quality. A candidate can closely resemble a reference while reversing a negation or getting a number wrong. Results also depend on the encoder and the reference: model biases and reference errors can affect the score. Combine metrics and inspect consequential errors rather than relying on one score. section 3.4.2 section 3.5


