liveBook · Manning

Evaluation dataset

As explained in Context Engineering

An evaluation set is a collection of test cases and expected or judged outcomes used to assess a system's performance.

An evaluation dataset is a curated collection of test cases used to measure an AI system’s performance, safety, and accuracy. For context-aware systems, cases should exercise how the system uses retrieval, memory, tools, and orchestration—not just whether it produces an answer. section 8.2.3

A useful dataset has several properties:

  • Representative: cases reflect the tasks the system is expected to handle. section 8.2.3
  • Stratified: cases are grouped by difficulty, domain, context source, failure mode, or user intent, so aggregate scores do not hide weak areas. section 8.2.3
  • Reviewed: human experts check cases for quality, ambiguity, and relevance. section 8.2.3
  • Versioned: changes are tracked, and evaluation cases are kept separate from prompt examples or tuning data. section 8.2.3
  • Adversarial and negative: cases test whether the system rejects misleading context, asks for missing information, or refuses when evidence is insufficient. section 8.2.3

Testing context handling

A negative case can supply stale memory or a poisoned instruction and check whether the system ignores or rejects it. Another can omit necessary information and check whether the system asks for it instead of proceeding without sufficient evidence. section 8.2.3

Cases can come from public datasets, domain experts, synthetic generation, or production use. Public datasets support general comparisons but may not match an application’s vocabulary or context policies. Synthetic cases scale quickly but need review for repetition and realism. Production-derived cases reflect real needs but require privacy review, anonymization, and filtering. section 8.2.3

In retrieval-augmented generation, rerun a frozen evaluation set when the corpus, chunking logic, or embedding model changes. Golden test sets can assess candidate indexes offline using retrieval precision, recall, and semantic similarity. Review coverage as domain vocabulary, multi-hop queries, and user intent evolve. Evaluation datasets also support regression tests for prompt changes and existing edge cases. section 3.2.5 section 9.3.5 section 6.2.2

How Context Engineering covers it

How deep it goes
Defines dataset criteria and sources, then gives operational checks for retrieval and prompt changes.
Examples
Uses stale memory, poisoned instructions, and missing information as negative cases.
What you can do after
Evaluate and understand dataset coverage, source trade-offs, and regression checks for AI systems.
How it connects
Connects retrieval-augmented generation, memory, tool use, and regression testing.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.