liveBook · Manning

Generator-critic loop

As explained in Designing AI Agents

A pattern in which a generator produces a candidate result and a critic evaluates it to guide revision or delivery.

A generator-critic loop separates production from review. A generator creates an artifact, a critic checks a specific version against criteria and evidence, and the generator revises it or delivers it when the review is satisfied. The canonical path is generate → critique → revise; further reviews are bounded by a stop policy. section 7.2

The key design question is what the critic can see that the generator did not. A different model or role alone does not guarantee independence: a new review frame, domain constraints, tests, source documents, or human judgment can expose assumptions that the original generation missed. Each finding should stay within the scope of its evidence. section 7.1.1 section 7.2.1

Reviewing a SQL query

A query may assume a join key is unique. Paraphrasing the query can preserve that assumption; checking schema constraints, sampled records, execution results, or join cardinality can reveal it. A parser checks syntax, while expected-row tests check a known behavior; a domain reviewer may still need to decide whether the result answers the business question. section 7.2.1

Build the review path around an explicit contract:

  • Specify the artifact version, review criteria, evidence sources, permitted verdicts, and maximum number of reviews. section 7.2 section 7.2.3
  • Collect evidence outside the critic and link findings to durable evidence identifiers. Retain the exact version inspected with each critique. section 7.2.3
  • Preserve candidate versions and select the best using domain evidence, such as regression results or a calibrated rubric. Another model’s preference alone is insufficient. section 7.2.3
  • Protect tests, rubrics, and thresholds from changes by the producing process; legitimate changes need a separate review path. section 7.2.4
Listing 7.4 A bounded Generator-Critic chain
@dataclass
class ReviewResult:
    artifact: str
    status: str
    history: list[ReviewStep] = field(default_factory=list)
 
 
def generator_critic(
    task,
    contract,
    generate,
    revise,
    critic,
    checks,
    select_best,
):
    artifact = generate(task)
    versions = [artifact]
    history = []
 
    for _ in range(contract.max_reviews): #A
        step = review_once(
            task,
            artifact,
            contract,
            critic,
            checks,
        )
        history.append(step)
        if step.critique.verdict == "accept":
            return ReviewResult(artifact, "accepted", history)
        if step.critique.verdict == "stop":
            best = select_best(task, versions, checks)
            return ReviewResult(best, "unresolved", history)
 
        artifact = revise(task, artifact, step.critique)
        versions.append(artifact) #B
 
    best = select_best(task, versions, checks) #C
    return ReviewResult(best, "budget_exhausted", history)

A bounded implementation records review history, handles accept and stop verdicts, and preserves versions for evidence-based selection. section 7.2.3

Reviews cost time and can rubber-stamp shared assumptions. Revisions can also oscillate, fixing one issue while reintroducing another. Detect repeated issue and artifact signatures, return the best known version, and hand unresolved trade-offs to a wider decision process. Calibrate model-based scores against known examples and record criterion-level findings. Measure confirmed-issue precision, quality change, oscillation, latency, and cost. section 7.2.4 section 7.2 section 2.3.2

Generator-critic improves the current artifact; it does not by itself authorize deployment or recover a failed execution. Adversarial review tests a contested claim through an independent search for defects, while an approval gate decides whether a proposed action may execute. section 8.4.1 section 7.6

How Designing AI Agents covers it

How deep it goes
Develops review contracts, evidence collection, versioned critiques, bounded iteration, best-version selection, and failure controls.
Examples
Uses a SQL join-cardinality review and a Python implementation with evidence records and bounded review history.
What you can do after
Build it from scratch with explicit verdicts, independent checks, version preservation, and a review budget.
How it connects
Distinguishes artifact review from adversarial review, approval gates, and bounded failure recovery.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.