liveBook · Manning

Evidence coverage

As explained in Designing AI Agents

The extent to which decisive claims in a result are linked to the evidence required to support them.

Evidence coverage is the extent to which decisive claims are linked to the evidence needed to support them. For an agent run, assess it alongside whether the result was accepted. Step count comes afterward: a short wrong answer and a long unsupported answer are both failures. section 5.1.5

Coverage also applies to architecture claims: it asks which parts of a claim have actually been tested. It differs from evidence class, which describes where evidence came from and which claims it can support. A test can be useful without supporting the entire claim. section 11.7

Testing a group-creation workflow

Revoking permission between stages and checking that the next call is blocked tests the connected system’s handling of state and authority. It does not prove that the intended salary group exists. That claim needs an authoritative readback of the tenant, template, and resulting group. After completion works, delayed responses, concurrent runs, and varied inputs can test whether completion and safety conditions still hold. section 11.7

Build an assessment around the claims and consequences:

  • Define the accepted result and the evidence required for its decisive claims. Add evidence coverage and accepted outcome to reasoning telemetry; a trace alone records a path, not a correct business decision. section 5.1.5
  • Use a representative, stratified workload, including ordinary, known-failure, edge, and high-risk cases. Separate conditions that may change the architecture rather than hiding them in one score. section 11.7
  • Set evidence requirements according to consequences and uncertainty, not a universal sample count. Event frequency, tolerated error, variance, workload strata, and evaluator reliability affect the testing needed. section 11.7
  • Keep quality, latency, cost, and unauthorized or duplicate effects visible separately when evaluating operation. section 11.7

How Designing AI Agents covers it

How deep it goes
Defines coverage and distinguishes evidence classes, then outlines workload stratification and risk-dependent testing without prescribing a universal sample count.
Examples
Uses a group-creation workflow to distinguish permission-revocation tests from authoritative readback and operating evidence.
What you can do after
Evaluate whether decisive claims and architecture components have been supported by appropriate tests.
How it connects
Connects evidence coverage to reasoning telemetry, accepted outcomes, evidence classes, and workload stratification.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.