liveBook · Manning

Benchmark

As explained in Retrieval Augmented Generation, The Foundational Ideas

A benchmark is a standardized evaluation task, dataset, or reference result used to compare system performance.

A benchmark is a standardized evaluation task, dataset, or reference result used to compare system performance. In retrieval and RAG, it turns an abstract quality claim into measurable evidence: whether the answer is correct, whether a relevant document appears in the top k, and how highly it ranks. section 3.2.1

Use benchmarks carefully rather than treating a score as meaningful by itself:

  • Compare against a structurally similar baseline so that training or knowledge-integration changes can be isolated. section 3.2.1
  • Check test-set separation; models that have seen evaluation examples during training can produce misleading results. section 3.2.1
  • Read several metrics together: accuracy measures correct answers, Recall@k measures whether a correct document appears in the top k, and MRR rewards high rank. section 3.2.1
  • Include operational measures such as inference time, parameter count, memory, and index size when selecting a production system. section 3.2.1
  • Match the benchmark's question distribution to real use: fact lookup, summary, reasoning, and unanswerable queries expose different retrieval behavior. section 14.4

A misleadingly strong result

A system can achieve perfect Recall@100 when the correct document appears at rank 50, while contributing only 0.02 to MRR. It has found the answer but buried it beyond the few documents passed to the reader. Likewise, a system optimized on fact_single-heavy data may score well yet fail on enterprise summary and reasoning queries. section 3.2.1 section 14.4

Table 14.3 Question-context taxonomy for RAG evaluation datasets
Class
Description
Example query
Retrieval difficulty
fact_single
Answer is one unit of information, directly stated in context
"What supply voltage should I use?"
Easiest: direct lexical/semantic match
summary
Answer requires multiple pieces from context
"Summarize the key findings."
Medium: must retrieve the broad context
reasoning
Answer must be inferred, not directly stated
"Has consumption increased over 5 years?"
Hardest: context doesn't contain the answer verbatim
unanswerable
Answer cannot be derived from the context
"Is a tomato a fruit or a vegetable?" (asked of a coffee machine patent)
Tests negative rejection

Use the question-context taxonomy to distinguish direct lookup, multi-part summary, inferred reasoning, and unanswerable evaluation cases. section 14.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains metric interpretation, baselines, splits, query distributions, and efficiency trade-offs, but does not detail benchmark implementation.
Examples
Uses Natural Questions, TriviaQA, BEIR, MTEB, HotpotQA, MS MARCO, and SQuAD as evaluation references.
What you can do after
Evaluate retrieval systems by comparing metrics, baselines, test splits, query types, and operational costs.
How it connects
Connects retrieval models, RAG systems, evaluation datasets, hybrid search, accuracy, Recall@k, and MRR.

Also explained in

Compare how 30 books explain benchmark →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.