liveBook · Manning

ROUGE

As explained in LLM Evaluation and Alignment, The Foundational Ideas

ROUGE is a family of automatic text-generation metrics that compares system output with reference text using overlapping units such as n-grams or subsequences.

A summary can cover an article accurately while using different words from a reference. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) compares generated text with human reference text using lexical overlap. Originally designed for summarization, it emphasizes recall: how much of the reference appears in the candidate. section 1.1.2 section 2.0

For ROUGE-N recall, count the matching reference n-grams and divide by the total reference n-grams. An n-gram is a sequence of n consecutive words. A match is capped by its count in each text, so repeating a word cannot create unlimited credit. Precision instead asks how much of the candidate appears in the reference; an F-measure can balance precision and recall. section 2.2.2 section 2.2.1 section 2.2.3

Counting ROUGE-1 and ROUGE-2

Reference: “Google announced new AI features for search.” Candidate: “Google revealed AI search capabilities.” Three of seven reference words match: Google, AI, and search. ROUGE-1 recall is 3/7, or about 0.429. None of the six reference word pairs match, so ROUGE-2 recall is 0. The different scores reveal how unigram overlap can coexist with little phrase overlap. section 2.2.2

Variants capture different kinds of overlap:

  • ROUGE-N counts matching consecutive n-grams; ROUGE-1 uses single words and ROUGE-2 uses word pairs. section 2.2.2
  • ROUGE-L uses the longest common subsequence: matching words must retain their relative order, but gaps are allowed. section 2.2.2
  • ROUGE-W weights consecutive subsequence matches more heavily; ROUGE-S counts ordered word pairs that need not be adjacent. section 2.2.2

Use ROUGE as a fast, repeatable signal, not a complete quality judgment. Lexical matching can miss synonyms and valid paraphrases, and overlap alone does not establish meaning or coherence. A long candidate can also improve recall while including incorrect information. Complement ROUGE with semantic metrics and human review when those distinctions matter. section 2.2.3 section 2.2.1 section 2.4

Monitoring a retrieval-augmented system

A nightly test suite can compare responses with references using ROUGE-L. A sudden score drop after a prompt change flags a change worth investigating; it does not by itself explain whether meaning or quality worsened. Follow up with targeted evaluation or human review. section 2.3.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains clipped n-gram recall, subsequence variants, and precision-recall balance with a worked calculation.
Examples
Calculates unigram and bigram recall for a short summary and describes nightly ROUGE-L regression testing.
What you can do after
Evaluate and understand reference overlap, choose complementary variants, and interpret score changes as prompts for further review.
How it connects
Connects lexical overlap to precision and recall, BLEU, longest common subsequence, and semantic similarity metrics.

Also explained in

Compare how 3 books explain ROUGE →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.