BERTScore evaluates generated text by comparing candidate and reference sentences through contextual semantic representations. It is useful for assessing summaries and translations when wording differs but meaning remains similar. section 12.2 section 12.2.3
The calculation follows four implementation-oriented stages:
- Represent candidate and reference tokens with pretrained BERT embeddings that reflect word context. section 12.2.3
- Compare token embeddings using cosine similarity and match each candidate token with the most similar reference token. section 12.2.3
- Aggregate token matches into precision and recall, then combine them into an F1 score. section 12.2.3
- Interpret the metric alongside task-specific benchmarks and human evaluation because numerical similarity does not fully represent output quality. section 12.6
Evaluating an article summary
A workflow downloads and parses an article, generates a summary, and uses the Evaluate library with BERTScore alongside BLEU and ROUGE. In the reported run, BERTScore produced precision 0.8524, recall 0.8710, and F1 0.8616. The high semantic score indicated that the candidate captured the reference's overall meaning despite weaker word-overlap scores. section 12.2.4
Table 12.1 Traditional evaluation metrics
|
Metric
|
Focus
|
Method
|
Limitations
|
|---|---|---|---|
|
BLEU
|
It primarily measures precision, the percentage of words in the machine- generated text that appear in the reference text.
|
It compares n-grams (word sequences) of the candidate translation with the reference translation and counts the matches.
|
It can miss the mark on semantic meaning because it doesn’t account for synonyms or the context of words. It also doesn’t handle word reordering well.
|
|
ROUGE
|
It is more recall oriented, focusing on the percentage of words from the reference text that appear in the generated text.
|
It has several variants, such as ROUGE-N, which compares n-grams, and ROUGE-L, which looks at the longest common subsequence.
|
Like BLEU, ROUGE can overlook semantic similarities and paraphrasing because it’s based on exact word matches.
|
|
BERTScore
|
It evaluates semantic similarity rather than relying on exact word matches.
|
It uses contextual embeddings from models such as BERT to represent the text and calculates the cosine similarity between these embeddings.
|
It can capture paraphrasing and semantic meaning better than BLEU and ROUGE because it considers each word’s context.
|
Compare BERTScore with BLEU and ROUGE by focus, method, and limitations. section 12.2
