liveBook · Manning

Recall

As explained in Retrieval Augmented Generation, The Foundational Ideas

Recall is the proportion of relevant items that a retrieval or classification system successfully returns.

A retrieval system can return plausible documents and still miss evidence needed for an answer. Recall measures coverage: of all relevant documents in the corpus, what fraction did the retriever return? For a list of k results, Recall@k measures that fraction within the top k. Precision@k asks the complementary question: what fraction of the returned documents are relevant? section 14.1.1

Five compliance documents

A financial services assistant returns five documents for a compliance query. Three are relevant, and the corpus contains four relevant documents in total. Recall@5 is 3/4 = 0.75; Precision@5 is 3/5 = 0.6. Returning more documents might recover the fourth, but it can also add irrelevant context. section 14.1.1 section 8.1

Figure 14.1 Precision and recall. Of the items returned by the retriever (the selected set), precision measures the fraction that are relevant. Of all relevant items in the corpus, recall measures the fraction that the retriever surfaced.

The selected and relevant sets make the denominator visible: recall divides their overlap by everything relevant, not by everything returned. section 14.1.1

Recall does not say where relevant documents appear in the ranking. A correct document at rank 50 counts as found in Recall@100, even though a reader receiving only the first few documents will miss it. Mean reciprocal rank (MRR) rewards an earlier first relevant result; NDCG@k accounts for the positions of multiple relevant results. A relevance label also cannot establish that a document contains the particular information a generator needs. section 3.2.1 section 14.1.1

Evaluation conventions matter. For a query with several relevant documents, Recall@k is the fraction retrieved. Some open-domain question-answering benchmarks instead score each question as 1 when a correct document appears anywhere in the top k, and 0 otherwise. Check which definition a reported result uses, and compare systems on an unseen test set rather than treating an isolated score as decisive. section 14.1.1 section 3.2.1

The useful recall target depends on what happens after retrieval:

  • With a reader that handles only a few passages, prioritize getting likely answer-bearing passages near the top. Fusion-in-Decoder can process a larger retrieved set and filter noise during decoding, shifting the retriever toward getting the evidence somewhere in that set; added passages can still increase noise and latency. section 5.3.1 section 5.7
  • For a query that misses relevant facets, RAG-Fusion runs multiple query variants to broaden coverage, then fuses their results. Measure Recall@k alongside precision and diversity: broader retrieval costs time and may bring in irrelevant documents. section 8.1 section 8.6
  • At index scale, compare recall with search latency and memory use. Approximate search can be worthwhile when a small recall loss makes retrieval substantially faster. section 2.2.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines Recall@k, works a numerical example, and examines ranking limitations; defers full ground-truth construction and end-to-end evaluation.
Examples
Uses a five-document compliance query and a rank-50 result to contrast coverage with ranking.
What it assumes
Defines precision and recall from relevant and retrieved document sets while using RAG retrieval as the application context.
What you can do after
Evaluate/understand retrieval coverage while checking ranking, reader capacity, latency, and noise before increasing the result count.
How it connects
Connects recall to precision, F1, MRR, NDCG, Fusion-in-Decoder, and multi-query retrieval.

Also explained in

Compare how 35 books explain recall →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.