liveBook · Manning

Iterative hypothesis testing

As explained in Designing AI Agents

An evaluation pattern in which successive passes propose hypotheses and test them against evidence.

Iterative hypothesis testing is an agent reasoning loop that uses external evidence to support or refute an active explanation. It is useful when an answer depends on external state and a bounded experiment can reduce uncertainty. Unlike a fixed pipeline, the loop changes direction when observations challenge its hypothesis. section 5.5 section 5.5.1

A disciplined investigation keeps explicit state across tool calls:

  • Represent candidate hypotheses and predict what observations would support or weaken each one. section 5.5
  • Choose a discriminating observation or bounded experiment, granting only the authority it needs. section 5.5
  • Preserve actual observations and errors, update the evidence, and state what changed after the test. section 5.5.1
  • Converge on an explanation, change direction, or stop when tests add no information, evidence remains ambiguous, or the budget is exhausted. section 5.5 section 5.5.1
Figure 5.5 Iterative Hypothesis Testing seeks discriminating evidence, updates the active explanation, and stops when the next cycle cannot justify its cost or authority.

The loop makes both evidence-driven convergence and the stopping branch explicit. section 5.5.1

The critical design question is what evidence would make the agent abandon its current hypothesis. Seeking only confirming evidence invites confirmation bias; confidence should not rise merely because the agent has spent more iterations. Repeating a test with different wording is stalled execution, not progress. A known transient failure may warrant a bounded deterministic retry; an unknown cause calls for investigation or escalation. section 5.5 section 5.5.1 section 2.3.2

Investigating a blank globe

A GIS renderer shows a blank globe, but the fault may lie upstream: reversed coordinate-axis order can pass through publishing without an error. Rather than render the same state again, check whether the client can retrieve a usable service, whether publishing exposed the expected layer, and whether processing preserved coordinate order. Service responses and a browser render provide observations from different stages. Evidence collection does not itself authorize a restart or other recovery action. section 5.5 section 5.9

Bound experiment permissions, runtime, output, and overall budget. A sandboxed code runner can enforce an allowlist, no shell, a pinned working directory, a timeout, and bounded output. If the next test cannot justify its cost or authority, stop with the evidence collected rather than continue indefinitely. section 5.5 section 5.5.1

How Designing AI Agents covers it

How deep it goes
Specifies hypothesis state, discriminating predictions, evidence updates, experiment permissions, budgets, and stopping rules.
Examples
Traces a blank GIS globe through service retrieval, publishing, and coordinate-axis processing checks.
What you can do after
Apply it to structure a bounded investigation and produce an evidence-rich handoff when a cause remains unresolved.
How it connects
Connects the scientific method and ReAct to explicit hypotheses, evidence updates, and stopping rules.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.