Recall measures coverage: how much of the relevant material a system finds or includes. For document retrieval, it is the number of relevant documents retrieved divided by the total number of relevant documents for the query. Low recall means relevant documents were missed. section 6.0 section 2.2.1
Precision asks a different question: how much of the retrieved or produced material is correct? A comprehensive output can have high recall while including incorrect information. Choose the emphasis for the task, or use an F-measure to balance precision and recall. section 2.2.1 section 2.2.3
In summary evaluation, ROUGE applies the same coverage idea to a reference summary: ROUGE-N recall divides the count of matching n-grams by the number of reference n-grams. An n-gram is a sequence of N words. Matches are limited by their counts in both summaries, so repetition cannot artificially inflate the score. section 2.2.2
Calculating ROUGE recall
Reference: “Google announced new AI features for search.” Candidate: “Google revealed AI search capabilities.” Three of the seven reference words match, giving ROUGE-1 recall of 3/7, or about 0.429. None of the six reference two-word sequences match, so ROUGE-2 recall is 0/6. section 2.2.2
A lexical recall score is not a complete measure of information coverage. ROUGE-N can miss paraphrases because it relies on word overlap; ROUGE also cannot assess meaning or coherence. Use complementary measures and interpret scores in the context of the task. section 2.2.2 section 2.2.3
Table 2.3 ROUGE metric variants and their use cases
|
Metric
|
What it means
|
Best used when
|
Limitations
|
|
ROUGE-N (N=1, 2)
|
N-gram recall. How many reference n-grams appear in the candidate?
|
Quick evaluations; comparing systems on the same datasets; need interpretable scores.
|
Misses paraphrasing; ROUGE-2 often zero for short texts.
|
|
ROUGE-L
|
Longest common subsequence (LCS). Word order preservation with gaps allowed
|
Evaluating restructured content, paraphrasing detection, and flexible word positioning matters.
|
Does not distinguish consecutive vs scattered matches.
|
|
ROUGE-W
|
Weighted LCS. Rewards consecutive matches more than scattered ones.
|
Evaluating fluency, phrase structure preservation, and distinguishing coherent from fragmented text.
|
More complex to implement; requires tuning the weighting function
|
|
ROUGE-S
|
Skip-bigram co-occurrence. Word pairs maintain order but not adjacency.
|
Long documents; capturing long-range dependencies; content where related concepts are separated.
|
Combinatorial explosion for long texts; may reward coincidental word pairs.
|
|
ROUGE-SU
|
Skip-bigram + unigram. ROUGE-S with unigram fallback.
|
Short texts where ROUGE-S might return zero; need robust scores across varying lengths,
|
Same limitations as ROUGE-S, plus potential noise from unigram.
|
Compare ROUGE variants and their different ways of matching reference content. section 2.2.2


































