A RAG assistant can score well on familiar fact lookups yet fail when users ask it to compare policies or explain a trend. An evaluation dataset is a collection of test cases used to measure performance under specified conditions. For end-to-end RAG evaluation, a case can include a query, context, reference answer, and question-type label; running the query through the system produces a response to score. Keep test cases separate from training data so memorization does not masquerade as performance. section 3.2.1 section 14.4.1 section 14.5
Choose and build cases to match the decision the evaluation must support:
- Use established QA benchmarks, such as Natural Questions, HotpotQA, MS MARCO, or SQuAD, to compare retrieval models. Do not assume their question mix represents an enterprise workload: single-fact questions dominate many such benchmarks. section 14.4
- For an application-specific set, generate questions from its own documents. Extract factual statements before turning them into questions, then use targeted prompts for summary and reasoning cases. Label each case so scores can be broken down by type; check generated answers and questions for errors. section 14.4.1
- Include unanswerable cases and retain difficult production failures. Remove cases that no longer reflect the corpus, and regenerate synthetic cases as it changes. section 14.4 section 15.6.4
Table 14.3 Question-context taxonomy for RAG evaluation datasets
|
Class
|
Description
|
Example query
|
Retrieval difficulty
|
|
fact_single
|
Answer is one unit of information, directly stated in
context
|
"What supply voltage should I use?"
|
Easiest: direct lexical/semantic match
|
|
summary
|
Answer requires multiple pieces from context
|
"Summarize the key findings."
|
Medium: must retrieve the broad context
|
|
reasoning
|
Answer must be inferred, not directly stated
|
"Has consumption increased over 5 years?"
|
Hardest: context doesn't contain the answer verbatim
|
|
unanswerable
|
Answer cannot be derived from the context
|
"Is a tomato a fruit or a vegetable?" (asked of
a coffee machine patent)
|
Tests negative rejection
|
The question-context taxonomy distinguishes direct facts, summaries, reasoning, and unanswerable questions. Use it to inspect coverage rather than relying on one aggregate score. section 14.4
Run each test query through the full system and score the resulting query, context, and response. Compare results by question type and by metric: a retrieval fix can improve the context while making the generated answer less faithful. Component tests isolate stages; end-to-end tests catch their interactions. A fixed set of previously failed queries supports regression checks before deployment. section 14.5 section 14.6 section 15.6.4
A weekly compliance-assistant evaluation
A mortgage compliance assistant used 300 cases: 120 fact lookups, 80 summaries, 60 reasoning questions, and 40 unanswerable questions. After a reranking fix, context relevance rose from 0.65 to 0.82, but faithfulness fell from 0.91 to 0.84. The set exposed a new generation problem that the retrieval gain alone would have hidden. section 14.6
Treat small score changes cautiously. Questions generated from the same passage share a source of difficulty, so they are not independent observations. When estimating uncertainty or comparing two runs on the same questions, resample by passage rather than by question. In the compliance example, the faithfulness decline held up under this check; a small answer-relevance increase did not. section 14.5.4








































