An evaluation suite is a collection of tasks designed to measure specific capabilities or behaviors of an AI agent. Each task defines inputs and success criteria; multiple trials can help estimate consistent behavior when outputs vary between runs. section 8.2.1
An evaluation harness runs the evaluations end to end. Graders score the agent’s transcript or its final environmental outcome. The transcript can include outputs, tool calls, intermediate results, and plans or reasoning artifacts that the system intentionally emits. section 8.2.1
Suite design and scoring serve different purposes:
- A capability suite tests challenging but solvable tasks the agent does not yet perform consistently. Its failures should reveal concrete engineering work, while its score leaves room for improvement. section 8.2.1
- Code-based graders are fast and reproducible but can reject valid variations. Model-based graders handle nuance but are nondeterministic and require calibration. Human graders support expert judgment and calibration but are costly and difficult to scale. section 8.2.1
Table 8.3 Comparison of grader types for AI agents across speed, cost, flexibility, and reproducibility
| Grader type | Advantages | Disadvantages |
Code-based |
Fast, cheap, reproducible |
Brittle, lacks nuance |
Model-based |
Flexible, scalable, handles open-ended tasks |
Non-deterministic, requires calibration |
Human |
Best alignment with expert judgment |
Expensive, slow, difficult to scale |
Compare grader types by speed, cost, flexibility, and reproducibility when choosing how to score suite tasks. section 8.2.1
Evaluation in a RAG pipeline
Continuous telemetry can feed regression traces into automated evaluation suites. Separately, offline evaluation checks candidate retrieval indexes against golden test sets before deployment. Coverage validation checks whether evaluation datasets reflect evolving vocabulary, multi-hop queries, and real-world user intent. section 9.3.5





















