liveBook · Manning

Hypothesis testing

As explained in Designing AI Agents

An analytical technique that evaluates candidate explanations or predictions against evidence or observations.

Hypothesis testing turns a suspicion into a bounded test plan. Argus can run only supplied sandbox checks; confirming a fault does not authorize recovery or allow production changes. section 5.5.4

Listing 5.14 Argus bug verification through hypothesis testing (`argus/reasoning.py`)
def verify_bug(self, suspicion: str, repo_path: str):
    tester = HypothesisTester(
        client=self.client,
        execute_fn=lambda cmd: run_in_sandbox(  #A
➥    cmd, repo_path),
        max_iterations=5,  #B
    )
    return tester.investigate(  #C
        f"Verify whether this is a real bug: {suspicion}"
    )

The verification method delegates a suspicion to a tester configured with sandbox execution and a five-iteration limit. section 5.5.4

How Designing AI Agents covers it

How deep it goes
Constrains investigation to five iterations and sandbox checks; confirming a fault does not authorize recovery.
Examples
Uses Argus to investigate whether a stated suspicion is a real bug.
What you can do after
Diagnose/debug with it by testing a suspicion through bounded sandbox checks.
How it connects
Links bug verification with sandboxing and controlled execution through execute_fn.

Also explained in

Compare how 8 books explain hypothesis testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.