liveBook · Manning

BERTScore

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A text-generation metric that compares semantic representations of candidate and reference text using contextual language-model embeddings.

BERTScore is a reference-based metric for comparing the meaning of generated text with a reference. Unlike lexical metrics such as BLEU and ROUGE, it uses contextual language-model embeddings to recognize similar meanings expressed in different words. section 3.0 section 3.2

Recognizing a paraphrase

Against the reference “people like foreign cars,” the candidate “consumers prefer imported cars” conveys a similar meaning despite using different words. A word-overlap metric can penalize this valid paraphrase; BERTScore compares learned semantic representations instead. section 3.0

BERTScore uses a contextual embedding for each token: the representation depends on the surrounding text. For example, “bank” can have a different representation near “money” than near “river.” BERTScore is a metric, not the BERT model itself; it can use other underlying encoders, including RoBERTa. section 3.2.2

To compare a candidate with a reference:

  • Encode both texts with a pretrained model to obtain contextual token embeddings. section 3.2.4
  • Calculate cosine similarity between every candidate token and every reference token. section 3.2.4
  • For precision, find each candidate token’s best reference match and average the similarities. For recall, reverse the direction: find each reference token’s best candidate match and average those similarities. section 3.2.4
  • Optionally weight matches by inverse document frequency (IDF), giving distinctive words more influence than common words. Baseline rescaling can also calibrate scores. section 3.2.4 section 3.2.5
Figure 3.7 Computing BERTScore. Given the reference and candidate, we compute embeddings and pairwise cosine similarity. The heatmap highlights the greedy matching, where each reference token is matched to its best candidate token. Optional importance weighting emphasizes less common words over common ones. And we will scale the score to be more evenly distributed between 0 and 1.

The token-similarity heatmap illustrates greedy matching between a candidate and a reference. section 3.2.4

Use BERTScore when valid paraphrases matter, such as in summarization or translation, and a suitable reference is available. It is more computationally expensive and less interpretable than lexical overlap metrics. Exact-match metrics remain useful when a changed word can make an output invalid, as in SQL extraction. section 2.3.6 section 3.1.1 section 3.4.1

A high similarity score is not a guarantee of factual accuracy or overall quality. A candidate can closely resemble a reference while reversing a negation or getting a number wrong. Results also depend on the encoder and the reference: model biases and reference errors can affect the score. Combine metrics and inspect consequential errors rather than relying on one score. section 3.4.2 section 3.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains contextual token embeddings, pairwise cosine similarity, greedy precision and recall, optional IDF weighting, and baseline rescaling.
Examples
Compares “consumers prefer imported cars” with “people like foreign cars” and works through greedy matching for a weather paraphrase.
What you can do after
Evaluate and understand semantic similarity scores while recognizing factual errors, reference dependence, and encoder bias.
How it connects
Connects contextual embeddings and cosine similarity to BLEU, ROUGE, and COMET as alternative evaluation approaches.

Also explained in

Compare how 3 books explain BERTScore →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.