Evaluation-driven development (EDD) makes evaluations a proactive driver of agent development. Teams define planned capabilities and their evaluation cases before the agent can reliably perform them, then improve the system in small, controlled steps until it performs well. section 8.2.1 section 11.4.1
A practical workflow combines capability development with regression protection:
- Define the task: specify its input, expected output or outcome, and relevant constraints. section 11.4.1
- Build evaluations around challenging but solvable cases and choose graders and metrics that make failures actionable. section 8.2.1
- Iterate on the agent, run capability and regression suites, and promote representative cases to the regression suite once the capability is consistently handled. section 8.2.1
- If a suite becomes saturated, add harder cases, stratify the benchmark, verify grader fairness, and remove ambiguous tasks. section 8.2.1
Capability evaluations measure skills the agent does not yet perform consistently; regression evaluations protect skills it already has and should target a pass rate near 100%. Individual passes are not enough: aggregate signals such as retrieval recall, latency, and tool use can reveal weak behavior across repeated runs. section 8.2.1
For coding agents, evaluate both the final artifact and the path used to produce it. Artifact checks include tests, static analysis, security scanners, type checking, build reproducibility, and code review. Path evidence includes context traces, tool-call logs, retrieved documents, prompt templates, and decision records. Metrics can be deterministic, qualitative, or structured LLM-as-judge assessments. section 11.4.1
Evaluating an API endpoint
For an agent implementing a small endpoint from a written specification, a deterministic metric can check whether the test suite passes. Qualitative assessment and LLM-as-judge metrics can complement direct checks by assessing properties such as maintainability, specification adherence, and edge-case handling. section 11.4.1
Table 8.3 Comparison of grader types for AI agents across speed, cost, flexibility, and reproducibility
| Grader type | Advantages | Disadvantages |
Code-based |
Fast, cheap, reproducible |
Brittle, lacks nuance |
Model-based |
Flexible, scalable, handles open-ended tasks |
Non-deterministic, requires calibration |
Human |
Best alignment with expert judgment |
Expensive, slow, difficult to scale |
Compare code-based, model-based, and human graders when selecting how to assess evaluation cases. section 8.2.1
