Human-in-the-loop
Human-in-the-loop is a system pattern in which people provide input, judgment, review, or intervention within an automated process.
Introduction to Generative AI, Second Edition
Human-in-the-loop is a system-design pattern in which people participate in monitoring, evaluating, correcting, or governing an AI system.
- How it is used
- Keeps people involved in evaluating AI outputs and controlling consequential actions while allowing lower-risk work to proceed automatically.
- How deep it goes
- Explains review, approval, evaluation, and ongoing monitoring; leaves exact risk thresholds and technical implementation unspecified.
- Examples
- Uses Amazon Bedrock Agents’ confirmation pause, cybersecurity analyst review, and human comparisons of dialogue responses.
- What you can do after
- Evaluate/understand when human review can reduce risk and why approval alone does not guarantee meaningful oversight.
Read how Introduction to Generative AI, Second Edition explains human-in-the-loop →
The Art of AI Product Development
Human evaluation is the assessment of model outputs by people using qualitative or task-specific judgments.
- How it is used
- Places human judgment in model evaluation and AI workflows where output quality, uncertain actions, or consequential decisions require review.
- How deep it goes
- Defines approval-based oversight and gives review triggers, evaluation trade-offs, and risk-based alternatives without prescribing an implementation.
- Examples
- Uses hiring recommendations, customer-request escalation, product-prioritization preferences, and human ratings of tone and creativity.
- What you can do after
- Apply it via risk-based review gates, human escalation, and evaluation checks while accounting for oversight cost and accountability.
Read how The Art of AI Product Development explains human-in-the-loop →
AI Agents in Action, Second Edition
Human evaluation is the assessment of system outputs or behavior by people applying judgment that may depend on context, policy, tone, or edge cases.
- How it is used
- Places human judgment in agent evaluation, support escalation, and approval of high-stakes actions.
- How deep it goes
- Specifies checkpoint triggers, reviewer context, waiting-state behavior, and fallback defenses; gives an escalation-tool example without a full approval implementation.
- Examples
- Uses email sending, fund transfers, and data deletion as approval cases, plus customer-support escalation for angry users or large refunds.
- What you can do after
- Apply it via risk-based checkpoints, contextual review, user-feedback capture, and an `escalate_to_human` tool.
Read how AI Agents in Action, Second Edition explains human-in-the-loop →
Generative AI in Action
Human evaluation is the assessment of an artifact or system by people using judgments, observations, surveys, or interviews.
- How it is used
- Keeps human judgment in LLM evaluation, generated-content review, and consequential decisions.
- How deep it goes
- Outlines criteria, evaluator selection, annotation, agreement analysis, and iteration, without specifying a software implementation.
- Examples
- Uses generated-code review, surveys and interviews, crowdsourced annotation, and critical machinery-repair instructions to illustrate differing oversight needs.
- What you can do after
- Practice/perform it by assessing outputs with rubrics, checking judgment reliability, and retaining human review or final control where risk warrants it.
Read how Generative AI in Action explains human-in-the-loop →
Starting Data Analytics with Generative AI and Python
Oversight is the monitoring and supervision of an AI system's performance, impacts, and compliance by responsible people or groups.
- How it is used
- Keeps people responsible for validating analytics outputs and monitoring deployed AI systems.
- How deep it goes
- Outlines review, testing, error monitoring, and corrective action without specifying a formal audit procedure.
- Examples
- Uses Portuguese customer-review sentiment classification with a financial-news model to show why error monitoring matters.
- What you can do after
- Practice/perform it by checking model outputs against reviews, monitoring errors, and escalating issues for corrective action.
Read how Starting Data Analytics with Generative AI and Python explains human-in-the-loop →
Machine Learning System Design
Judgments or preference signals supplied by people about the quality or desirability of system outputs.
- How it is used
- Uses human judgments to assess output quality before deployment and to decide whether model improvements justify release regressions.
- How deep it goes
- Describes ratings, pairwise comparisons, and release review, but gives no detailed assessor protocol.
- Examples
- Uses benchmark-query relevance ratings, old-versus-new search comparisons, and a model update that fixes three errors while introducing two.
- What you can do after
- Practice/perform human review of output quality and release tradeoffs while accounting for its cost and limited throughput.
Read how Machine Learning System Design explains human-in-the-loop →
Build an AI Agent (From Scratch)
Human evaluation is the direct assessment of system outputs by professional graders or domain experts.
- How it is used
- Keeps people in the loop to authorize sensitive agent tool calls or assess agent outputs.
- How deep it goes
- Implements synchronous callback approval and session-backed pause-and-resume; contrasts human grading with scalable evaluators without specifying a human-grading procedure.
- Examples
- Uses a file-deletion approval flow, pending tool calls, and a resumed agent response after confirmation.
- What you can do after
- Build it from scratch: gate dangerous tools with a callback, then pause and resume approval through saved session state.
Read how Build an AI Agent (From Scratch) explains human-in-the-loop →
AI-Powered Search
A judgment is a human assessment of the relevance of a search result to a query or information need.
- How it is used
- Defines relevance labels and corrected examples used to create training data for search ranking and question answering.
- How deep it goes
- Explains binary and multi-class labeling with concrete datasets, but stops short of a full reviewer protocol or inter-rater analysis.
- Examples
- Uses movie-query labels, model-generated question answers, corrected golden-set rows, and post-click behavior as evaluation evidence.
- What you can do after
- Apply it by labeling query-document pairs, correcting model outputs, and preparing higher-quality data for ranking or answer training.
AI Agents and Applications
- How it is used
- Routes consequential or uncertain agent decisions to a person before a response or action is finalized.
- How deep it goes
- Describes escalation triggers, approval, and feedback logging, but stops short of a concrete human-review implementation.
- Examples
- Uses itinerary approval, travel disruptions, accessibility requests, financial transactions, and medical recommendations as review cases.
- What you can do after
- Operate/configure it by escalating selected cases for approval and logging decisions to improve future automated handling.
Read how AI Agents and Applications explains human-in-the-loop →
"Looks Good to Me"
The human capacity to evaluate software decisions using nuance, context, history, and broader situational understanding.
- How it is used
- Reserves code-review attention for contextual decisions while automation handles repetitive checks.
- How deep it goes
- Explains contextual decisions through upgrade and variable-scope examples, but stops short of prescribing a formal judgment framework.
- Examples
- Uses library upgrades, removed service features, variable declarations, and AI-assisted pull-request reviews.
- What you can do after
- Evaluate software decisions, validate AI feedback, and make final code-review decisions using context and nuance.
Digital Twins in Action
Human oversight is the supervision and authorization of an autonomous system's decisions and actions by people.
- How it is used
- Sets decision authority for digital twins and agents whose recommendations or actions can affect physical operations.
- How deep it goes
- Defines three authority tiers and describes approval before commands, but does not specify an operator-interface or approval-service implementation.
- Examples
- Uses maintenance prioritization, equipment throttling, sensor recalibration, and an Open Policy Agent irrigation rule to distinguish authority levels.
- What you can do after
- Operate/configure it by assigning authority tiers, enforcing action constraints, and requiring approval before high-risk physical commands.
Read how Digital Twins in Action explains human-in-the-loop →
Data-Driven Cybersecurity
Human oversight is the review, supervision, or intervention by people in an automated system's operation and decisions.
- How it is used
- Positions human oversight as a safeguard for AI-driven security decisions and a governance check on cybersecurity performance.
- How deep it goes
- Explains human validation and regular assessment as oversight practices, without prescribing a detailed approval workflow.
- Examples
- Illustrates mistaken DDoS alerts from traffic surges and potentially disruptive access revocation based on behavioral data.
- What you can do after
- Practice/perform review of AI findings while assessing whether cybersecurity measures remain effective and aligned with organizational objectives.
Read how Data-Driven Cybersecurity explains human-in-the-loop →
Financial AI in Practice
Human oversight is the involvement of people in reviewing, approving, or supervising system decisions and operation.
- How it is used
- Requires human approval for high-risk generated outputs and governance review of financial models before deployment.
- How deep it goes
- Specifies human clicks and approval checkpoints, but gives no detailed review procedure.
- Examples
- Uses flagged financial posts, generated financial advice, credit-model approval, and investment signals requiring verification.
- What you can do after
- Configure human approval for flagged posts and high-risk outputs, and include governance review before model deployment.
Read how Financial AI in Practice explains human-in-the-loop →
Effective Conversational AI
Human review is the examination of system outputs by people who assess their accuracy and contextual appropriateness.
- How it is used
- Controls conversational AI risk by keeping people responsible for reviewing or approving LLM outputs.
- How deep it goes
- Distinguishes periodic retroactive sampling from beforehand approval and combines automated scoring with human verification.
- Examples
- Uses weekly response sampling, copy editing, dialogue rewriting, and LLM-assisted evaluation of grounded and contextually relevant answers.
- What you can do after
- Operate conversational AI with human approval, monitor failures, and verify response accuracy and contextual relevance.
Read how Effective Conversational AI explains human-in-the-loop →
First-Party Data Activation
A human-in-the-loop model is an AI system or workflow in which people participate in reviewing, approving, or directing model outputs.
- How it is used
- Controls generative marketing outputs through human review and approval before production.
- How deep it goes
- Recommends differentiated review and sampling, but stops short of detailing roles and workflows.
- Examples
- Uses campaign targeting, campaign assets, and one-to-one personalized content as review cases.
- What you can do after
- Apply it via approval processes for campaigns and sampling procedures for individualized content.
Read how First-Party Data Activation explains human-in-the-loop →
Building Reliable AI Systems
Human supervision of an AI system's outputs, decisions, and operation.
- How it is used
- Enforces human validation where AI errors could have serious professional consequences or complex logic requires careful review.
- How deep it goes
- Specifies legal review safeguards and agent approval steps, but gives no detailed human-review workflow implementation.
- Examples
- Uses fabricated legal citations, complex code logic, and agent execution checkpoints as review cases.
- What you can do after
- Apply human review before legal outputs reach court and retain careful review for complex code logic.
Read how Building Reliable AI Systems explains human-in-the-loop →
Grokking Statistics
Human in the loop is a system pattern in which people participate in reviewing, labeling, guiding, or validating automated processing.
- How it is used
- Frames human review as a risk-sensitive safeguard for automated predictions and as labor supporting model development.
- How deep it goes
- Explains the pattern through industry-risk comparisons and labeling workflows, but stops short of implementation procedures.
- Examples
- Contrasts Netflix recommendations with defense targeting and describes annotation by workers, doctors, and lawyers.
- What you can do after
- Operate predictive systems by planning human review, collecting new data, and updating and evaluating models as environments change.
Related concepts
- evaluatoralternative to · An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.
- AI agentused by · An AI agent is a computational system that uses artificial intelligence to interpret information, select actions, and pursue goals through interaction with an environment.
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- automationcontrasts with · Automation is the execution of tasks or processes by machines or software with reduced direct human intervention.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.