liveBook · Manning

Human-in-the-loop

As explained in Designing AI Systems

Human review is the inspection and validation of an item by a person before it is accepted for an intended purpose.

Human-in-the-loop review puts a person at a decision point that automation should not resolve alone. In production AI systems, review can gate a consequential action before execution or supply judgments that calibrate evaluation and validate test data. section 6.7.3 section 8.3.4

A human gate intercepts a high-stakes tool call before it reaches the external system. The platform stores the pending action, notifies an appropriate approver, and returns a holding response. Execution resumes after explicit approval; a timeout can expire the request or escalate it, according to policy. section 6.7.3

Listing 6.25 Human approval gate configuration
platform.guardrails.add_policy(
    name="high-stakes-approval",
    type="human_gate",
    rules=[
        {
            "tool": "healthcare.clinical.record_diagnosis",
            "approval_role": "physician",    #A
            "timeout_hours": 24,
            "on_timeout": "expire_and_notify_user"
        },
        {
            "tool": "healthcare.billing.process_refund",
            "condition": "amount > 500",    #B
            "approval_role": "billing_supervisor",
            "timeout_hours": 48
        },
        {
            "tool": "healthcare.scheduling.book_procedure",
            "condition": "procedure_category == 'surgical'",
            "approval_role": "scheduling_coordinator",
            "timeout_hours": 4,
            "on_timeout": "escalate_to_manager"    #C
        }
    ]
)

The policy assigns approver roles and timeouts to diagnosis, refund, and procedure-booking actions; conditions limit some gates to higher-risk cases. section 6.7.3

The gate belongs in the platform or client, not merely in the tool transport. MCP recommends human approval for tool invocations but leaves enforcement to the client. Approval complements other guardrails, including permission checks and action-consistency policies. section 6.4.4 section 6.7.3

Human evaluation serves a different purpose: reviewers label responses rather than approve actions. Because review is expensive and less scalable than automated scoring, a small production sample can provide ground-truth labels to calibrate automated scorers and LLM judges. A layered example uses automated scores on all traffic, model judges on 5–20%, and human review on 1–5%. section 7.6.1

Listing 8.6 Annotation queue for human review
queue = platform.experiments.create_annotation_queue(
    name="intake-quality-review",
    workflow_id="patient-intake",
    rubric=[       #A
        {"name": "correctness", "type": "categorical",
         "options": ["correct", "partially_correct",
                    "incorrect"]},
        {"name": "helpfulness", "type": "numeric",
         "min": 0.0, "max": 1.0},
        {"name": "add_to_dataset", "type": "boolean",    #B
         "description": "Should this be added to the "
                       "eval dataset?"},
    ],
    routing_rules=[
        {"condition": "score.helpfulness < 0.5",
         "source": "model_judge"},     #C
        {"condition": "tag contains 'guardrail_triggered'"},
        {"condition": "random_sample", "rate": 0.02},   #D
    ],
    reviewers=["sarah@healthfirst.com",
               "dr.patel@healthfirst.com"],
)

An annotation queue routes low-scoring, guardrail-triggered, and randomly sampled traces to reviewers using a rubric. section 8.3.4

Reviewers inspect the input, output, spans, and existing scores, then annotate against the rubric. Their scores are marked HUMAN and can be compared with automated judgments to detect calibration drift. Review can also validate production traces before they become evaluation test cases, preventing unsuitable examples from polluting the dataset. section 8.3.4 section 8.3.1

How Designing AI Systems covers it

How deep it goes
Details approval and timeout flows, policy configuration, annotation routing, and reviewer scoring.
Examples
Illustrates physician approval of diagnoses, supervisor review of large refunds, and rubric-based review of patient-intake traces.
What you can do after
Operate/configure approval gates and annotation queues with role-based routing, timeouts, sampling, and review rubrics.
How it connects
Connects behavioral guardrails and tool permissions with annotation queues, evaluation datasets, and LLM-as-judge scoring.

Also explained in

Compare how 28 books explain human-in-the-loop →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.