A benchmark is a standardized evaluation task, dataset, or reference result used to compare system performance. In retrieval and RAG, it turns an abstract quality claim into measurable evidence: whether the answer is correct, whether a relevant document appears in the top k, and how highly it ranks. section 3.2.1
Use benchmarks carefully rather than treating a score as meaningful by itself:
- Compare against a structurally similar baseline so that training or knowledge-integration changes can be isolated. section 3.2.1
- Check test-set separation; models that have seen evaluation examples during training can produce misleading results. section 3.2.1
- Read several metrics together: accuracy measures correct answers, Recall@k measures whether a correct document appears in the top k, and MRR rewards high rank. section 3.2.1
- Include operational measures such as inference time, parameter count, memory, and index size when selecting a production system. section 3.2.1
- Match the benchmark's question distribution to real use: fact lookup, summary, reasoning, and unanswerable queries expose different retrieval behavior. section 14.4
A misleadingly strong result
A system can achieve perfect Recall@100 when the correct document appears at rank 50, while contributing only 0.02 to MRR. It has found the answer but buried it beyond the few documents passed to the reader. Likewise, a system optimized on fact_single-heavy data may score well yet fail on enterprise summary and reasoning queries. section 3.2.1 section 14.4
Table 14.3 Question-context taxonomy for RAG evaluation datasets
|
Class
|
Description
|
Example query
|
Retrieval difficulty
|
|
fact_single
|
Answer is one unit of information, directly stated in
context
|
"What supply voltage should I use?"
|
Easiest: direct lexical/semantic match
|
|
summary
|
Answer requires multiple pieces from context
|
"Summarize the key findings."
|
Medium: must retrieve the broad context
|
|
reasoning
|
Answer must be inferred, not directly stated
|
"Has consumption increased over 5 years?"
|
Hardest: context doesn't contain the answer verbatim
|
|
unanswerable
|
Answer cannot be derived from the context
|
"Is a tomato a fruit or a vegetable?" (asked of
a coffee machine patent)
|
Tests negative rejection
|
Use the question-context taxonomy to distinguish direct lookup, multi-part summary, inferred reasoning, and unanswerable evaluation cases. section 14.4





























