A summary can cover an article accurately while using different words from a reference. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) compares generated text with human reference text using lexical overlap. Originally designed for summarization, it emphasizes recall: how much of the reference appears in the candidate. section 1.1.2 section 2.0
For ROUGE-N recall, count the matching reference n-grams and divide by the total reference n-grams. An n-gram is a sequence of n consecutive words. A match is capped by its count in each text, so repeating a word cannot create unlimited credit. Precision instead asks how much of the candidate appears in the reference; an F-measure can balance precision and recall. section 2.2.2 section 2.2.1 section 2.2.3
Counting ROUGE-1 and ROUGE-2
Reference: “Google announced new AI features for search.” Candidate: “Google revealed AI search capabilities.” Three of seven reference words match: Google, AI, and search. ROUGE-1 recall is 3/7, or about 0.429. None of the six reference word pairs match, so ROUGE-2 recall is 0. The different scores reveal how unigram overlap can coexist with little phrase overlap. section 2.2.2
Variants capture different kinds of overlap:
- ROUGE-N counts matching consecutive n-grams; ROUGE-1 uses single words and ROUGE-2 uses word pairs. section 2.2.2
- ROUGE-L uses the longest common subsequence: matching words must retain their relative order, but gaps are allowed. section 2.2.2
- ROUGE-W weights consecutive subsequence matches more heavily; ROUGE-S counts ordered word pairs that need not be adjacent. section 2.2.2
Use ROUGE as a fast, repeatable signal, not a complete quality judgment. Lexical matching can miss synonyms and valid paraphrases, and overlap alone does not establish meaning or coherence. A long candidate can also improve recall while including incorrect information. Complement ROUGE with semantic metrics and human review when those distinctions matter. section 2.2.3 section 2.2.1 section 2.4
Monitoring a retrieval-augmented system
A nightly test suite can compare responses with references using ROUGE-L. A sudden score drop after a prompt change flags a change worth investigating; it does not by itself explain whether meaning or quality worsened. Follow up with targeted evaluation or human review. section 2.3.3


