An evaluation dataset is a collection of representative user requests that can be run repeatedly against an agent. Across many scenarios, it makes outputs, component behavior, and trajectories observable—not just whether an answer looks right, but whether tools worked and the agent took a reasonable path. section 10.2.1
Choose cases from sources that serve different needs:
- Public benchmarks provide standardized comparisons, but strong scores may not reflect product use. section 10.2.2
- Custom cases reflect expected tasks; include different difficulty levels, domains, and edge cases. A small core set can grow over time. section 10.2.2
- Production requests reflect real use patterns, though they often lack ground-truth labels. section 10.2.2
Table 10.2 Types of evaluation datasets
| Type |
Description |
Pros |
Cons |
|---|---|---|---|
| Benchmark |
Standardized public evaluation sets (e.g., HotpotQA, GAIA, and SWE-bench) |
Comparable across systems; validated difficulty and quality |
May diverge from real use; risk of overfitting to the benchmark |
| Custom |
Internally constructed evaluation sets designed to reflect real scenarios |
Tailored to product needs; domain-specific evaluation |
High build cost |
| Production |
Real user requests and feedback collected in production |
Most realistic evaluation; reflects real use patterns |
Often lacks ground-truth labels |
Compare benchmark, custom, and production datasets by their advantages and limitations before choosing cases. section 10.2.2
Use cases as a deployment gate
When a prompt, policy, or flow changes, run a small set of roughly 20–50 core cases in premerge CI and check for regressions. After merging, run a larger set in staging. An automated gate can block deployment when the overall pass rate falls below a chosen threshold. section 10.4.1
Keep the dataset current. Review operational data, analyze failures, update the error taxonomy and rubric or metrics, then add cases that capture emerging patterns. A fixed test set will not keep pace with changing users, environments, and external tools. section 10.4.2
