liveBook · Manning

Evaluation receipt

As explained in Designing AI Agents

An evaluation receipt is a record of a model's training and generator versions, held-out workload, evaluation results, and release decision.

An evaluation receipt records the evidence and decision for an architecture proposal. It is separate from the Pattern Selection Card: the card proposes a composition, while the receipt records whether the comparison justified accepting it provisionally, rejecting it, or reopening the decision. This separation keeps a plausible design from being mistaken for an accepted one. section 10.6 section 10.11

Bind the receipt to an operating point and workload, and retain evidence for the baseline comparison, seam tests, external acceptance, and risk gates. A candidate earns its added mechanisms by correcting a measured deficit on the same workload; local component success alone does not establish continuity across seams. section 10.6 section 10.11 section 10.9

Listing 10.1 An architecture hypothesis and its evaluation receipt (`examples/ch10/composition_contracts.py`)
from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class ArchitectureHypothesis:  #A
    decision_id: str
    task_id: str
    baseline: str
    observed_deficit: str
    proposal: tuple[str, ...]
    assumption_evidence: tuple[str, ...]
    seam_contracts: tuple[str, ...]
    acceptance_gates: tuple[str, ...]
    removal_condition: str
 
 
@dataclass(frozen=True)
class EvaluationReceipt:  #B
    decision_id: str
    operating_point: str
    workload_version: str
    evidence_refs: tuple[str, ...]
    baseline_failed: bool
    proposal_passed: bool
    seam_tests_passed: bool
    external_acceptance_passed: bool
    risk_gates_passed: bool
 
 
def decide(receipt: EvaluationReceipt) -> str:
    if not receipt.evidence_refs:
        return "REJECT: no evidence attached"
    if not receipt.baseline_failed:  #C
        return "REJECT: baseline already meets the gates"
    if not all((
        receipt.proposal_passed,
        receipt.seam_tests_passed,
        receipt.external_acceptance_passed,
        receipt.risk_gates_passed,
    )):
        return "REJECT: proposal did not pass all required gates"
    return "ACCEPT_PROVISIONALLY"  #D

The data boundary separates an architecture hypothesis from its evaluation receipt and represents the evidence and gate results used for a decision. section 10.6

Learned routing

For a learned router, preserve the training and generator versions, held-out workload boundary, route-selection results, parameter fidelity, clarification behavior, and release decision. Schema validity alone is not enough: a route can select the wrong function or carry an invented parameter. Authority remains a separate gate when routes expose different effects or privileges. section 10.8.2

Keep the receipt alongside later operating evidence so changes in judgment remain traceable. Reopen the decision when changes to the model, tools, workload, sources, authority, or cost invalidate the operating point. If the same contract passes without a mechanism, the architecture can be simplified while preserving the evidence. section 11.7 section 10.9

How Designing AI Agents covers it

How deep it goes
Defines receipt fields and decision gates, then extends the record to held-out evaluation for a learned router.
Examples
Uses a learned router evaluated for route selection, parameter fidelity, and clarification behavior on a held-out workload.
What you can do after
Evaluate a candidate architecture using recorded evidence and gates, and reopen the decision when its operating point changes.
How it connects
Connects architecture evaluation to baseline comparison, seam tests, external acceptance, and risk gates.

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.