ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a family of reference-overlap metrics for generated summaries and translations. It emphasizes how much reference content the system output captures, while also reporting precision and F-measure for a broader comparison. section 12.2.2 section 12.2
Common ROUGE variants measure different forms of surface overlap:
- ROUGE-N compares matching n-grams, including single words and longer sequences. section 12.2.2
- ROUGE-L uses the longest common subsequence to reflect shared word order and structure. section 12.2.2
- Treat scores as evidence of lexical overlap rather than complete measures of quality: synonyms, paraphrases, fluency, word order, and stop-word or stemming choices can affect results. section 12.2.2 section 12.2.4
Interpreting a summary evaluation
An automated article-summary pipeline compares a generated summary with a reference summary using the Evaluate library. Reported results include ROUGE-1 of 0.3463, ROUGE-2 of 0.0961, ROUGE-L of 0.1645, and ROUGE-Lsum of 0.2684. The pattern indicates moderate word overlap, weaker phrase overlap, and limited shared sequence structure; compare it with semantic metrics and human judgment before making a production decision. section 12.2.4
Table 12.1 Traditional evaluation metrics
|
Metric
|
Focus
|
Method
|
Limitations
|
|---|---|---|---|
|
BLEU
|
It primarily measures precision, the percentage of words in the machine- generated text that appear in the reference text.
|
It compares n-grams (word sequences) of the candidate translation with the reference translation and counts the matches.
|
It can miss the mark on semantic meaning because it doesn’t account for synonyms or the context of words. It also doesn’t handle word reordering well.
|
|
ROUGE
|
It is more recall oriented, focusing on the percentage of words from the reference text that appear in the generated text.
|
It has several variants, such as ROUGE-N, which compares n-grams, and ROUGE-L, which looks at the longest common subsequence.
|
Like BLEU, ROUGE can overlook semantic similarities and paraphrasing because it’s based on exact word matches.
|
|
BERTScore
|
It evaluates semantic similarity rather than relying on exact word matches.
|
It uses contextual embeddings from models such as BERT to represent the text and calculates the cosine similarity between these embeddings.
|
It can capture paraphrasing and semantic meaning better than BLEU and ROUGE because it considers each word’s context.
|
Compare ROUGE with BLEU and BERTScore by focus, calculation method, and limitations. section 12.2
