liveBook · Manning

Evidence

As explained in Designing AI Agents

Information that supports or substantiates a claim or decision.

Evidence is information that supports a claim or decision. In an agent system, it can establish whether an action achieved its intended effect, justify a change in autonomy, or connect a failure to the decisions that produced it. A successful tool call alone does not establish the business outcome. section 6.1.2 section 9.5.6

Define acceptance evidence before treating an action as complete: records, receipts, or probes should establish the result and support recovery. An action contract can declare acceptance labels, while registered post-checks implement them for a particular tool. Keep execution attempts separate because an external outcome may remain uncertain. section 6.1.2 section 10.3

A returned HTTP 202, queued job, or locally successful deployment command is not necessarily acceptance. Persist state outside the model context, preserve accepted work, and verify uncertain effects before replanning the affected pending work. section 6.4.6

Evidence also governs commitment at two timescales. Within a session, freeze the proposal and relevant resource version, run cheaper checks, and revalidate the same proposal before committing. Across sessions, accumulated results can support staged trust escalation. A dry run produces evidence without changing the target system, but it can still observe a different state from the live call. Checks add latency, so use the full chain where consequences justify it; the blast-radius ceiling remains fixed. section 9.4.1

Figure 9.3 Progressive Commitment. Freeze the proposal, stage cheaper checks, and revalidate the same proposal before commit.

Shows the sequence from a frozen proposal through cheaper checks to revalidation before commit. section 9.4.1

Protect evidence used for review. A critic should cite a criterion or evidence item for each finding, and changes to tests or acceptance criteria should follow a separate review path. For governance, connect decisions, execution, and outcomes in a causal record; relevant artifacts can include policy versions, approvals, traces, incident history, and the basis for autonomy changes. No single pattern proves compliance. section 7.2.4 section 9.9

Investigating excess compensation

If a coupon tool call succeeds but the customer receives more than authorized, trace the proposal back to the messages, case limit, earlier coupons, and policy decision. An excessive proposal points toward input or planning repair; an admitted proposal may also reveal a gate or task-state failure. Turn the incident into a regression case that denies the excessive proposal, and test a valid request too. Link new traces to the incident and compare the gate decision with the business outcome before staged deployment. section 9.5.6

How Designing AI Agents covers it

How deep it goes
Details acceptance records, staged checks, trust escalation, and incident investigation, including latency and state-change limitations.
Examples
Uses dry runs, deployment return statuses, critic findings, and an excess-coupon incident to illustrate different evidence needs.
What you can do after
Diagnose/debug with evidence by tracing a proposal through policy decisions to business outcomes and creating regression cases.
How it connects
Connects evidence to action contracts, progressive commitment, blast radius control, observability, and acceptance criteria.

Also explained in

Compare how 8 books explain evidence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.