liveBook · Manning

Evidence class

As explained in Designing AI Agents

An evidence class is a category describing the kind of evidence available for assessing an architectural claim.

An evidence class describes the kind of evidence available for an architectural claim and limits what that evidence can support. Assess the claim alongside its workload and operating conditions before deciding what architectural action it warrants. The class alone does not determine the size of the change. section 11.1 section 11.1.1

Five classes support different claims:

  • Illustrative design exercise: A constructed scenario can expose a design decision or test whether a method asks useful questions, but cannot establish production performance. section 11.1.1
  • Repository-executed case: Rerunnable code, fixtures, and expected outcomes support claims about a mechanism and its implementation, not necessarily another model, organization, or data distribution. section 11.1.1
  • Recorded benchmark: A preserved result package supports claims about its recorded comparison, but omitted inputs or generation steps may prevent an independent rerun. section 11.1.1
  • Attributed practitioner account: Direct experience can support a reported failure shape and design rationale, but not audited performance, transferability, or reconstructed details. section 11.1.1
  • Public production report: An organization's report can reveal its stated deployment context and disclosed results, but does not by itself establish causality or transferability. section 11.1.1

Evidence class is not evidence coverage. Class describes provenance and the claims evidence can support; coverage describes which parts of an architecture claim were actually tested. A deterministic fixture can establish a mechanism cleanly, while a production report can reveal operating constraints the fixture cannot reach. A new deployment still needs evidence from its own workload. section 11.1.1 section 11.10 section 11.7

Testing a salary-import proposal

A dispatch test can show that a gate admits a permitted call and rejects an altered protected argument, yet miss a stale receipt supplied by an adapter. A handoff test follows values through state, approval, and import admission. An authoritative readback checks the resulting group; operating tests add delays, concurrent runs, and input variation. Track quality, latency, cost, and unauthorized or duplicate effects separately. section 11.7

There is no universal evidence threshold. Failure frequency and cost, tolerated error, observed variance, workload strata, and evaluator reliability affect the testing needed. Scoped evidence can justify keeping, repairing, recomposing, simplifying, or retiring a decision. section 11.7 section 11.10

Figure 11.1 Five evidence classes bind evidence to a scoped claim before it can justify keeping, repairing, recomposing, simplifying, or retiring an architecture decision.

Use the figure to trace the five evidence classes through a scoped claim to an architecture action. section 11.1.1

How Designing AI Agents covers it

How deep it goes
Defines five classes with supported claims and limits, then distinguishes evidence provenance from test coverage.
Examples
Uses a salary-import proposal to contrast dispatch, handoff, readback, and operating tests.
What you can do after
Evaluate architecture claims against their evidence limits and identify further workload-specific testing.
How it connects
Connects evidence classes to test coverage, reproducibility, transferability, and operational evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.