liveBook · Manning

Evaluation dataset

As explained in Retrieval Augmented Generation, The Foundational Ideas

An evaluation set is a collection of examples used to measure system performance under specified evaluation conditions.

A RAG assistant can score well on familiar fact lookups yet fail when users ask it to compare policies or explain a trend. An evaluation dataset is a collection of test cases used to measure performance under specified conditions. For end-to-end RAG evaluation, a case can include a query, context, reference answer, and question-type label; running the query through the system produces a response to score. Keep test cases separate from training data so memorization does not masquerade as performance. section 3.2.1 section 14.4.1 section 14.5

Choose and build cases to match the decision the evaluation must support:

  • Use established QA benchmarks, such as Natural Questions, HotpotQA, MS MARCO, or SQuAD, to compare retrieval models. Do not assume their question mix represents an enterprise workload: single-fact questions dominate many such benchmarks. section 14.4
  • For an application-specific set, generate questions from its own documents. Extract factual statements before turning them into questions, then use targeted prompts for summary and reasoning cases. Label each case so scores can be broken down by type; check generated answers and questions for errors. section 14.4.1
  • Include unanswerable cases and retain difficult production failures. Remove cases that no longer reflect the corpus, and regenerate synthetic cases as it changes. section 14.4 section 15.6.4
Table 14.3 Question-context taxonomy for RAG evaluation datasets
Class
Description
Example query
Retrieval difficulty
fact_single
Answer is one unit of information, directly stated in context
"What supply voltage should I use?"
Easiest: direct lexical/semantic match
summary
Answer requires multiple pieces from context
"Summarize the key findings."
Medium: must retrieve the broad context
reasoning
Answer must be inferred, not directly stated
"Has consumption increased over 5 years?"
Hardest: context doesn't contain the answer verbatim
unanswerable
Answer cannot be derived from the context
"Is a tomato a fruit or a vegetable?" (asked of a coffee machine patent)
Tests negative rejection

The question-context taxonomy distinguishes direct facts, summaries, reasoning, and unanswerable questions. Use it to inspect coverage rather than relying on one aggregate score. section 14.4

Run each test query through the full system and score the resulting query, context, and response. Compare results by question type and by metric: a retrieval fix can improve the context while making the generated answer less faithful. Component tests isolate stages; end-to-end tests catch their interactions. A fixed set of previously failed queries supports regression checks before deployment. section 14.5 section 14.6 section 15.6.4

A weekly compliance-assistant evaluation

A mortgage compliance assistant used 300 cases: 120 fact lookups, 80 summaries, 60 reasoning questions, and 40 unanswerable questions. After a reranking fix, context relevance rose from 0.65 to 0.82, but faithfulness fell from 0.91 to 0.84. The set exposed a new generation problem that the retrieval gain alone would have hidden. section 14.6

Treat small score changes cautiously. Questions generated from the same passage share a source of difficulty, so they are not independent observations. When estimating uncertainty or comparing two runs on the same questions, resample by passage rather than by question. In the compliance example, the faithfulness decline held up under this check; a small answer-relevance increase did not. section 14.5.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Details benchmark selection, corpus-grounded generation, question-type coverage, recurring tests, and passage-grouped uncertainty estimates.
Examples
Uses QA benchmarks, generated query-context-answer triples, and a 300-case mortgage compliance evaluation.
What it assumes
Assumes familiarity with RAG queries, retrieved context, and responses; explains how to assemble and use their evaluation cases.
What you can do after
Build it from scratch using corpus passages, then apply it to diagnose RAG changes and maintain deployment checks.
How it connects
Connects evaluation datasets to retrieval metrics, answer faithfulness, synthetic question generation, integration tests, and regression tests.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.