liveBook · Manning

Evaluation-driven development

As explained in Context Engineering

A software engineering paradigm that makes evaluations the proactive driver of development.

Evaluation-driven development (EDD) makes evaluations a proactive driver of agent development. Teams define planned capabilities and their evaluation cases before the agent can reliably perform them, then improve the system in small, controlled steps until it performs well. section 8.2.1 section 11.4.1

A practical workflow combines capability development with regression protection:

  • Define the task: specify its input, expected output or outcome, and relevant constraints. section 11.4.1
  • Build evaluations around challenging but solvable cases and choose graders and metrics that make failures actionable. section 8.2.1
  • Iterate on the agent, run capability and regression suites, and promote representative cases to the regression suite once the capability is consistently handled. section 8.2.1
  • If a suite becomes saturated, add harder cases, stratify the benchmark, verify grader fairness, and remove ambiguous tasks. section 8.2.1

Capability evaluations measure skills the agent does not yet perform consistently; regression evaluations protect skills it already has and should target a pass rate near 100%. Individual passes are not enough: aggregate signals such as retrieval recall, latency, and tool use can reveal weak behavior across repeated runs. section 8.2.1

For coding agents, evaluate both the final artifact and the path used to produce it. Artifact checks include tests, static analysis, security scanners, type checking, build reproducibility, and code review. Path evidence includes context traces, tool-call logs, retrieved documents, prompt templates, and decision records. Metrics can be deterministic, qualitative, or structured LLM-as-judge assessments. section 11.4.1

Evaluating an API endpoint

For an agent implementing a small endpoint from a written specification, a deterministic metric can check whether the test suite passes. Qualitative assessment and LLM-as-judge metrics can complement direct checks by assessing properties such as maintainability, specification adherence, and edge-case handling. section 11.4.1

Table 8.3 Comparison of grader types for AI agents across speed, cost, flexibility, and reproducibility
Grader type Advantages Disadvantages

Code-based

Fast, cheap, reproducible

Brittle, lacks nuance

Model-based

Flexible, scalable, handles open-ended tasks

Non-deterministic, requires calibration

Human

Best alignment with expert judgment

Expensive, slow, difficult to scale

Compare code-based, model-based, and human graders when selecting how to assess evaluation cases. section 8.2.1

How Context Engineering covers it

How deep it goes
Details task definition, metric categories, regression-suite promotion, and remedies for evaluation saturation.
Examples
Uses a coding agent implementing an API endpoint from a written specification, with its test suite as a deterministic check.
What you can do after
Practice evaluation-driven development by defining task criteria, iterating on agent behavior, and checking for regressions.
How it connects
Connects context engineering, regression testing, LLM-as-judge, and observability to agent quality.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.