An evaluation dataset is a curated collection of test cases used to measure an AI system’s performance, safety, and accuracy. For context-aware systems, cases should exercise how the system uses retrieval, memory, tools, and orchestration—not just whether it produces an answer. section 8.2.3
A useful dataset has several properties:
- Representative: cases reflect the tasks the system is expected to handle. section 8.2.3
- Stratified: cases are grouped by difficulty, domain, context source, failure mode, or user intent, so aggregate scores do not hide weak areas. section 8.2.3
- Reviewed: human experts check cases for quality, ambiguity, and relevance. section 8.2.3
- Versioned: changes are tracked, and evaluation cases are kept separate from prompt examples or tuning data. section 8.2.3
- Adversarial and negative: cases test whether the system rejects misleading context, asks for missing information, or refuses when evidence is insufficient. section 8.2.3
Testing context handling
A negative case can supply stale memory or a poisoned instruction and check whether the system ignores or rejects it. Another can omit necessary information and check whether the system asks for it instead of proceeding without sufficient evidence. section 8.2.3
Cases can come from public datasets, domain experts, synthetic generation, or production use. Public datasets support general comparisons but may not match an application’s vocabulary or context policies. Synthetic cases scale quickly but need review for repetition and realism. Production-derived cases reflect real needs but require privacy review, anonymization, and filtering. section 8.2.3
In retrieval-augmented generation, rerun a frozen evaluation set when the corpus, chunking logic, or embedding model changes. Golden test sets can assess candidate indexes offline using retrieval precision, recall, and semantic similarity. Review coverage as domain vocabulary, multi-hop queries, and user intent evolve. Evaluation datasets also support regression tests for prompt changes and existing edge cases. section 3.2.5 section 9.3.5 section 6.2.2








































