Evaluation assesses an agent’s results and changes against evidence appropriate to the decision. Place it beside the change: a review of an artifact, a recovery attempt, a proposed capability, or a lesson considered for reuse each needs different checks. section 7.7 section 7.1.1
Use the cheapest signal that can decide the current question, then broaden the evidence when a failure can escape that signal’s scope. A well-formed output is not necessarily correct, and a verifier covers only the checks encoded in it. Semantic review is stronger when its criteria and evidence give it a perspective independent of the generator. section 7.7 section 7.1.1
Figure 7.6 The evaluation evidence stack. Structural checks, deterministic verifiers, traces, invariants, semantic review, human evidence, and real outcomes cover progressively different questions and feedback latencies.
The evidence stack distinguishes questions and feedback latencies across structural checks, execution evidence, review, and real outcomes. section 7.7
Match the evidence to the change being evaluated:
- For artifact review, combine a semantic rubric with deterministic evidence where available. section 7.7
- For recovery, use an external verifier, regression checks, and the rollback result. section 7.7
- For a reusable skill, evaluate it in isolation and alongside other skills, then run a controlled trial. Different applicability conditions may justify keeping more than one version. section 7.7 section 7.4.2
- For a reusable lesson, keep the generated heuristic as a candidate until independent evaluation admits it; later outcomes help determine whether it actually helped. section 4.5.3 section 7.7
Evaluate across runs as well as within them. A safeguard can pass its local check while overlapping with other rules or increasing system latency. Track cost, latency, state change, and human effort alongside task success; do not let an aggregate score hide a hard regression. section 7.0 section 7.9
Golden SQL as a production canary
For selected natural-language questions, compare generated SQL and executed results with manually authored golden SQL and expected data. Run the checks in development and as production canaries. A failure identifies what went wrong; traces and result artifacts help localize the first deviation before choosing an architectural response. Update the golden set as workloads and failure modes change. section 11.4.3




























