liveBook · Manning

Recall

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Document recall is the proportion of relevant documents that an information-retrieval system retrieves for a query.

Recall measures coverage: how much of the relevant material a system finds or includes. For document retrieval, it is the number of relevant documents retrieved divided by the total number of relevant documents for the query. Low recall means relevant documents were missed. section 6.0 section 2.2.1

Precision asks a different question: how much of the retrieved or produced material is correct? A comprehensive output can have high recall while including incorrect information. Choose the emphasis for the task, or use an F-measure to balance precision and recall. section 2.2.1 section 2.2.3

In summary evaluation, ROUGE applies the same coverage idea to a reference summary: ROUGE-N recall divides the count of matching n-grams by the number of reference n-grams. An n-gram is a sequence of N words. Matches are limited by their counts in both summaries, so repetition cannot artificially inflate the score. section 2.2.2

Calculating ROUGE recall

Reference: “Google announced new AI features for search.” Candidate: “Google revealed AI search capabilities.” Three of the seven reference words match, giving ROUGE-1 recall of 3/7, or about 0.429. None of the six reference two-word sequences match, so ROUGE-2 recall is 0/6. section 2.2.2

A lexical recall score is not a complete measure of information coverage. ROUGE-N can miss paraphrases because it relies on word overlap; ROUGE also cannot assess meaning or coherence. Use complementary measures and interpret scores in the context of the task. section 2.2.2 section 2.2.3

Table 2.3 ROUGE metric variants and their use cases
Metric
What it means
Best used when
Limitations
ROUGE-N (N=1, 2)
N-gram recall. How many reference n-grams appear in the candidate?
Quick evaluations; comparing systems on the same datasets; need interpretable scores.
Misses paraphrasing; ROUGE-2 often zero for short texts.
ROUGE-L
Longest common subsequence (LCS). Word order preservation with gaps allowed
Evaluating restructured content, paraphrasing detection, and flexible word positioning matters.
Does not distinguish consecutive vs scattered matches.
ROUGE-W
Weighted LCS. Rewards consecutive matches more than scattered ones.
Evaluating fluency, phrase structure preservation, and distinguishing coherent from fragmented text.
More complex to implement; requires tuning the weighting function
ROUGE-S
Skip-bigram co-occurrence. Word pairs maintain order but not adjacency.
Long documents; capturing long-range dependencies; content where related concepts are separated.
Combinatorial explosion for long texts; may reward coincidental word pairs.
ROUGE-SU
Skip-bigram + unigram. ROUGE-S with unigram fallback.
Short texts where ROUGE-S might return zero; need robust scores across varying lengths,
Same limitations as ROUGE-S, plus potential noise from unigram.

Compare ROUGE variants and their different ways of matching reference content. section 2.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Works through n-gram recall calculations and introduces document recall as a retrieval relevance metric.
Examples
Calculates unigram and bigram recall for a candidate summary of a search-feature announcement.
What you can do after
Evaluate lexical coverage using ROUGE-N recall and recognize when overlap misses paraphrases.
How it connects
Connects recall to precision, F-measure, ROUGE, and retrieval-augmented generation.

Also explained in

Compare how 35 books explain recall →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.