liveBook · Manning

Evaluation

As explained in Designing AI Agents

Evaluation is the systematic assessment of a result against criteria for quality, correctness, or suitability.

Evaluation assesses an agent’s results and changes against evidence appropriate to the decision. Place it beside the change: a review of an artifact, a recovery attempt, a proposed capability, or a lesson considered for reuse each needs different checks. section 7.7 section 7.1.1

Use the cheapest signal that can decide the current question, then broaden the evidence when a failure can escape that signal’s scope. A well-formed output is not necessarily correct, and a verifier covers only the checks encoded in it. Semantic review is stronger when its criteria and evidence give it a perspective independent of the generator. section 7.7 section 7.1.1

Figure 7.6 The evaluation evidence stack. Structural checks, deterministic verifiers, traces, invariants, semantic review, human evidence, and real outcomes cover progressively different questions and feedback latencies.

The evidence stack distinguishes questions and feedback latencies across structural checks, execution evidence, review, and real outcomes. section 7.7

Match the evidence to the change being evaluated:

  • For artifact review, combine a semantic rubric with deterministic evidence where available. section 7.7
  • For recovery, use an external verifier, regression checks, and the rollback result. section 7.7
  • For a reusable skill, evaluate it in isolation and alongside other skills, then run a controlled trial. Different applicability conditions may justify keeping more than one version. section 7.7 section 7.4.2
  • For a reusable lesson, keep the generated heuristic as a candidate until independent evaluation admits it; later outcomes help determine whether it actually helped. section 4.5.3 section 7.7

Evaluate across runs as well as within them. A safeguard can pass its local check while overlapping with other rules or increasing system latency. Track cost, latency, state change, and human effort alongside task success; do not let an aggregate score hide a hard regression. section 7.0 section 7.9

Golden SQL as a production canary

For selected natural-language questions, compare generated SQL and executed results with manually authored golden SQL and expected data. Run the checks in development and as production canaries. A failure identifies what went wrong; traces and result artifacts help localize the first deviation before choosing an architectural response. Update the golden set as workloads and failure modes change. section 11.4.3

How Designing AI Agents covers it

How deep it goes
Explains evidence selection and limitations across reflection patterns, including admission checks, delayed outcomes, and cross-run review.
Examples
Uses accumulated build safeguards and golden SQL comparisons to illustrate system-level costs and production failure localization.
What you can do after
Diagnose failures with traces and evaluation results, then use localized evidence to guide architecture changes.
How it connects
Connects evaluation to reflection, regression testing, semantic review, rollback, and experience replay.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.