Human-in-the-loop
Human-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.
Introduction to Generative AI, Second Edition
Human-in-the-loop is a system-design pattern in which people participate in monitoring, evaluating, correcting, or governing an AI system.
- How it is used
- Places people in review, correction, and approval roles as AI systems assist with work or take actions.
- How deep it goes
- Explains review and approval patterns with concrete applications, while leaving implementation details unspecified.
- Examples
- Illustrates agent approval with Amazon Bedrock Agents and response comparison with human evaluations of Llama 2-Chat.
- What you can do after
- Evaluate oversight choices by considering task risk, reviewer expertise, and the limits of human judgment.
Read how Introduction to Generative AI, Second Edition explains human-in-the-loop →
AI Governance
An approach in which a person participates in an AI system’s operation, such as by reviewing or authorizing its outputs or actions.
- How it is used
- Provides review and approval checkpoints for sensitive AI outputs and actions, limiting unreviewed autonomy in production workflows.
- How deep it goes
- Details review triggers, approval thresholds, reviewer authority and supporting controls, while recognizing that oversight of rapid black-box agents remains unsolved.
- Examples
- Uses a fabricated clinical note and an insurance claim workflow to illustrate rubber-stamping and targeted review.
- What you can do after
- Operate/configure review workflows by selecting approval gates, giving reviewers authority and using override patterns to identify system weaknesses.
The Art of AI Product Development
Human evaluation is the assessment of model outputs by people using qualitative or task-specific judgments.
- How it is used
- Uses human judgment to evaluate model outputs and oversee AI decisions or agent actions where reliability, risk, or user needs warrant it.
- How deep it goes
- Explains oversight levels, review triggers, evaluation trade-offs, and safeguards without detailing a complete implementation.
- Examples
- Uses hybrid language-model evaluation, human review of hiring recommendations, and agent requests for missing interview data.
- What you can do after
- Apply human review to model evaluation and AI workflows by selecting oversight points and weighing review effort against risk.
Read how The Art of AI Product Development explains human-in-the-loop →
AI Agents in Action, Second Edition
Human evaluation is the assessment of system outputs or behavior by people applying judgment that may depend on context, policy, tone, or edge cases.
- How it is used
- Places human judgment in agent approval, escalation, output evaluation, and feedback loops.
- How deep it goes
- Details checkpoint triggers, reviewer context, waiting state, and failure safeguards; outlines rubric design and validation against human judgment.
- Examples
- Uses approval for sending email or transferring funds and a support-agent escalation tool for upset customers and large refunds.
- What you can do after
- Apply risk-based approval checkpoints and human review alongside automated checks and other safeguards.
Read how AI Agents in Action, Second Edition explains human-in-the-loop →
Architecting for Autonomy
A supervisory autonomy pattern in which a human monitors autonomous decisions and intervenes when behavior exceeds expected boundaries.
- How it is used
- Places human authority at bounded decision surfaces and distinguishes embedded review from supervisory oversight.
- How deep it goes
- Details trigger classes, handover requirements, validation, reviewer capacity, and risk-based selection between oversight postures.
- Examples
- Uses secured lending applications with low confidence, excessive exposure, or inconsistent documentation to illustrate human review.
- What you can do after
- Apply risk-based oversight by defining escalation triggers, testing handovers, and assessing reviewer capacity.
Read how Architecting for Autonomy explains human-in-the-loop →
LLM Evaluation and Alignment, The Foundational Ideas
A human preference signal is an evaluation signal derived from people's comparative or scalar judgments of output quality.
- How it is used
- Uses human quality judgments to evaluate open-ended outputs and oversee automated decisions.
- How deep it goes
- Details annotation guidance, sampling, golden-set monitoring, agreement metrics, and escalation procedures.
- Examples
- Uses a customer service chatbot response and crowd-labeling instructions to illustrate quality judgments.
- What you can do after
- Evaluate automated judges using human audit samples, agreement metrics, and a golden dataset.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains human-in-the-loop →
Generative AI in Action
Human evaluation is the assessment of an artifact or system by people using judgments, observations, surveys, or interviews.
- How it is used
- Adds human judgment and intervention to AI evaluation, content review, and critical decisions.
- How deep it goes
- Outlines evaluation criteria, evaluator selection, annotation, agreement analysis, and iterative improvement, alongside production oversight considerations.
- Examples
- Uses generated-code review, surveys, interviews, and crowdsourced annotation to illustrate human assessment.
- What you can do after
- Evaluate and understand AI outputs using human criteria, evaluator guidelines, and interrater agreement.
Read how Generative AI in Action explains human-in-the-loop →
Starting Data Analytics with Generative AI and Python
Oversight is the monitoring and supervision of an AI system's performance, impacts, and compliance by responsible people or groups.
- How it is used
- Supports human validation of automated analytics and oversight of AI performance and impacts.
- How deep it goes
- Outlines periodic expert review, automated issue flagging, and corrective responsibility without specifying a detailed review protocol.
- Examples
- Uses a Portuguese customer-review classifier trained on financial news and an AI-use policy with monitoring and reporting provisions.
- What you can do after
- Practice periodic review of data quality and AI outcomes alongside automated checks and error monitoring.
Read how Starting Data Analytics with Generative AI and Python explains human-in-the-loop →
Machine Learning System Design
Judgments or preference signals supplied by people about the quality or desirability of system outputs.
- How it is used
- Uses human judgments to evaluate outputs before deployment and weigh model improvements against regressions at release.
- How deep it goes
- Compares rating and preference judgments, discusses cost and throughput, and identifies review responsibilities and safeguards for automatic updates.
- Examples
- Uses search-result ratings and comparisons, image-search reasonableness judgments, and a release with three corrected errors and two new ones.
- What you can do after
- Evaluate whether human ratings or comparisons suit a quality or release decision, accounting for review costs and regressions.
Read how Machine Learning System Design explains human-in-the-loop →
Build a Multi-Agent System (from Scratch)
Human-in-the-loop is a design pattern in which people participate in or oversee stages of an automated process.
- How it is used
- Adds human input and oversight to agent tasks involving ambiguity, costly results, or risks of cascading errors.
- How deep it goes
- Details tool-based input, result approval, and stepwise supervision, including control tradeoffs and multi-agent extension requirements.
- Examples
- Uses restaurant selection to illustrate clarification and a critical bug-fix plan to illustrate approval and supervision.
- What you can do after
- Builds HITL support from scratch using a human-input tool, a result-approval gate, and a supervised task handler.
Read how Build a Multi-Agent System (from Scratch) explains human-in-the-loop →
Build an AI Agent (From Scratch)
Human evaluation is the direct assessment of system outputs by professional graders or domain experts.
- How it is used
- Uses human approval to gate sensitive agent actions and human evaluation to assess agent outputs.
- How deep it goes
- Implements synchronous callbacks and asynchronous pause-and-resume approval, while comparing human output evaluation with automated approaches.
- Examples
- Uses file deletion to demonstrate approval, rejection, and later resumption with a recorded decision.
- What you can do after
- Builds an approval workflow from callbacks, session state, pending tool calls, and confirmation decisions.
Read how Build an AI Agent (From Scratch) explains human-in-the-loop →
Designing AI Systems
Human review is the inspection and validation of an item by a person before it is accepted for an intended purpose.
- How it is used
- Uses human judgment to gate high-stakes tool actions and calibrate AI response evaluation.
- How deep it goes
- Details approval and timeout flows, policy configuration, annotation routing, and reviewer scoring.
- Examples
- Illustrates physician approval of diagnoses, supervisor review of large refunds, and rubric-based review of patient-intake traces.
- What you can do after
- Operate/configure approval gates and annotation queues with role-based routing, timeouts, sampling, and review rubrics.
AI-Powered Search
A judgment is a human assessment of the relevance of a search result to a query or information need.
- How it is used
- Refines model-generated guesses into human-labeled training data and supplies relevance judgments for ranking.
- How deep it goes
- Defines a correction workflow and five label states, but provides no code that can generate the golden set automatically.
- Examples
- Uses question-answering model guesses in a CSV and movie-document relevance labels for example queries.
- What you can do after
- Practice manual labeling by classifying guesses, correcting answers, and refining a training set.
AI Agents and Applications
- How it is used
- Routes uncertain or consequential agent requests to people before responses or actions are finalized.
- How deep it goes
- Outlines escalation triggers and feedback steps without implementing an approval mechanism.
- Examples
- Uses travel itineraries, disruption queries, accessibility requests, and high-value transactions as review cases.
- What you can do after
- Evaluate and understand which agent cases merit human approval and how review decisions can inform improvements.
Read how AI Agents and Applications explains human-in-the-loop →
"Looks Good to Me"
The human capacity to evaluate software decisions using nuance, context, history, and broader situational understanding.
- How it is used
- Reserves code-review attention for contextual decisions while automation handles repetitive checks.
- How deep it goes
- Explains contextual decisions through upgrade and variable-scope examples, but stops short of prescribing a formal judgment framework.
- Examples
- Uses library upgrades, removed service features, variable declarations, and AI-assisted pull-request reviews.
- What you can do after
- Evaluate software decisions, validate AI feedback, and make final code-review decisions using context and nuance.
Digital Twins in Action
Human oversight is the supervision and authorization of an autonomous system's decisions and actions by people.
- How it is used
- Governs digital twin decisions and production changes when physical actions or their consequences warrant human approval.
- How deep it goes
- Defines authority tiers and approval boundaries, but stops short of implementing a human approval workflow.
- Examples
- Contrasts lighting, maintenance scheduling, and reactor pressure decisions, and illustrates an irrigation constraint with a policy rule.
- What you can do after
- Evaluate/understand when human approval is appropriate and distinguish it from constraints on automated actions.
Read how Digital Twins in Action explains human-in-the-loop →
Vibe Engineering
The monitoring and direction of an automated system by people who retain authority to identify errors and correct its course.
- How it is used
- Keeps agentic IDE work aligned with intent by having people approve, review, and correct agent actions.
- How deep it goes
- Explains selective approval and supervision costs, then contrasts human feedback with automated observability for unattended agents.
- Examples
- Uses file approvals, command validation, and the trade-off between thirty minutes saved and twenty minutes of supervision.
- What you can do after
- Operate agentic IDEs by calibrating approval policies to task risk and supervising novel or destructive actions.
Data-Driven Cybersecurity
Human oversight is the review, supervision, or intervention by people in an automated system's operation and decisions.
- How it is used
- Positions human oversight as a safeguard for AI-driven security decisions and a governance check on cybersecurity performance.
- How deep it goes
- Explains human validation and regular assessment as oversight practices, without prescribing a detailed approval workflow.
- Examples
- Illustrates mistaken DDoS alerts from traffic surges and potentially disruptive access revocation based on behavioral data.
- What you can do after
- Practice/perform review of AI findings while assessing whether cybersecurity measures remain effective and aligned with organizational objectives.
Read how Data-Driven Cybersecurity explains human-in-the-loop →
Context Engineering
Human oversight is the involvement of people in monitoring, reviewing, or controlling consequential AI decisions and actions.
- How it is used
- Provides structured human review and approval for consequential or uncertain AI actions.
- How deep it goes
- Details risk-based routing, review, trace recording, evaluation feedback, and operational limits, alongside a basic interactive loop.
- Examples
- Uses customer-support escalation, a collaborative story-writing loop, and approval before database migrations or financial transactions.
- What you can do after
- Apply HITL by routing proposed actions to reviewers, recording decisions, and resuming workflows after approval.
Financial AI in Practice
Human oversight is the involvement of people in reviewing, approving, or supervising system decisions and operation.
- How it is used
- Places human approval at consequential decision points in financial AI workflows.
- How deep it goes
- Explains approval checkpoints and their organizational rationale, without specifying an implementation procedure.
- Examples
- Illustrates oversight through post moderation, financial-advice approval, and credit-model governance review.
- What you can do after
- Evaluate where human approval and verification fit into financial AI workflows.
Read how Financial AI in Practice explains human-in-the-loop →
Effective Conversational AI
Human review is the examination of system outputs by people who assess their accuracy and contextual appropriateness.
- How it is used
- Controls conversational AI risk by keeping people responsible for reviewing or approving LLM outputs.
- How deep it goes
- Distinguishes periodic retroactive sampling from beforehand approval and combines automated scoring with human verification.
- Examples
- Uses weekly response sampling, copy editing, dialogue rewriting, and LLM-assisted evaluation of grounded and contextually relevant answers.
- What you can do after
- Operate conversational AI with human approval, monitor failures, and verify response accuracy and contextual relevance.
Read how Effective Conversational AI explains human-in-the-loop →
Understanding AI
Human oversight is the involvement of people in monitoring, evaluating, and controlling an AI system's outputs or behavior.
- How it is used
- Places human judgment between AI-generated outputs and consequential action, especially when models encounter external instructions.
- How deep it goes
- Provides a practical checklist for reviewing outputs but does not specify a detailed evaluation framework or technical control mechanism.
- Examples
- Uses external documents, websites, emails, trusted sources, output review, and high-stakes decisions as oversight scenarios.
- What you can do after
- Evaluate AI-generated outputs critically before using them, particularly when decisions carry significant consequences.
First-Party Data Activation
A human-in-the-loop model is an AI system or workflow in which people participate in reviewing, approving, or directing model outputs.
- How it is used
- Uses human review and approval to manage AI-generated marketing outputs before they enter production.
- How deep it goes
- Recommends review or sampling by output type, but defers detailed roles and workflows.
- Examples
- Contrasts campaign-level targeting and assets with one-to-one personalized content that may require sampling.
- What you can do after
- Apply it by establishing review and approval processes, including sampling where personalized outputs are difficult to review individually.
Read how First-Party Data Activation explains human-in-the-loop →
Building Specialized AI Systems
Human-in-the-loop is a system design approach that incorporates human review or intervention into an automated data or model workflow.
- How it is used
- Uses human verification to improve synthetic labels in an automated training-dataset workflow.
- How deep it goes
- Details automated generation and subsequent verification, but does not specify reviewer criteria or approval procedures.
- Examples
- Uses analyst reports, generated summaries, prompt templates, and a DataFrame summary column as concrete workflow artifacts.
- What you can do after
- Apply it via automated summary generation followed by human verification before using outputs for training.
Read how Building Specialized AI Systems explains human-in-the-loop →
Building Reliable AI Systems
Human supervision of an AI system's outputs, decisions, and operation.
- How it is used
- Enforces human validation where AI errors could have serious professional consequences or complex logic requires careful review.
- How deep it goes
- Specifies legal review safeguards and agent approval steps, but gives no detailed human-review workflow implementation.
- Examples
- Uses fabricated legal citations, complex code logic, and agent execution checkpoints as review cases.
- What you can do after
- Apply human review before legal outputs reach court and retain careful review for complex code logic.
Read how Building Reliable AI Systems explains human-in-the-loop →
Grokking Statistics
Human in the loop is a system pattern in which people participate in reviewing, labeling, guiding, or validating automated processing.
- How it is used
- Frames human review as a risk-sensitive safeguard for automated predictions and as labor supporting model development.
- How deep it goes
- Explains the pattern through industry-risk comparisons and labeling workflows, but stops short of implementation procedures.
- Examples
- Contrasts Netflix recommendations with defense targeting and describes annotation by workers, doctors, and lawyers.
- What you can do after
- Operate predictive systems by planning human review, collecting new data, and updating and evaluating models as environments change.
AI Model Evaluation
Human review is the assessment of model outputs or decisions by human evaluators.
- How it is used
- Uses human ratings to validate LLM judges and human review to support high-impact evaluation decisions.
- How deep it goes
- Identifies validation and review roles but does not detail a human-review workflow.
- What you can do after
- Evaluate judge reliability by comparing its outputs with human ratings.
Designing AI Agents
An approach in which a person intervenes in an automated process by supplying information, editing output, or judging a result.
- How it is used
- Uses people to supply information, edit output, or judge results within automated processes.
- How deep it goes
- Distinguishes broad human intervention from pre-execution approval, without detailing implementation of HITL itself.
- Examples
- Lists supplying missing information, editing a draft, and judging a result as human interventions.
Also covered in
- AI Agents for Offensive SecurityIn early access (MEAP)
- How Large Language Models Work
- Learn AI Data EngineeringIn early access (MEAP)
- The AI Pocket Book
- Software Testing with Generative AI
- Data Storytelling with Altair and AI
- Delivering Machine Learning Projects: From design to deployment
- AI for Everyday IT
- Akka in Action, Second Edition
- Elixir in Action, Third Edition
- Agent Design PatternsIn early access (MEAP)
- Learn AI-Assisted Python Programming, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- AI agentAn AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
- contractA contract is an agreed specification of the conditions, expectations, rights, or obligations that govern behavior or interactions between parties or software components.
Related concepts
- LLM-as-a-judgealternative to · LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
- AI agentused by · An AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
- AI Actconfigured by · The AI Act is a European Union regulation establishing harmonised, risk-based rules and obligations for artificial-intelligence systems and the actors who develop, provide, or use them.
- automationcontrasts with · Automation is the execution of tasks or processes by machines or software with reduced direct human intervention.
Questions about human-in-the-loop
What is human-in-the-loop?
Human-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.
Which Manning books explain human-in-the-loop?
Introduction to Generative AI, Second Edition, AI Governance, The Art of AI Product Development, AI Agents in Action, Second Edition, Architecting for Autonomy, LLM Evaluation and Alignment, The Foundational Ideas, Generative AI in Action, Starting Data Analytics with Generative AI and Python and 20 more.
What should I know before learning human-in-the-loop?
Manning books that teach human-in-the-loop usually introduce language model, fine-tuning, AI agent and contract first.
What is the difference between human-in-the-loop and LLM-as-a-judge?
Human-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process. LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
What is human-in-the-loop also called?
Human-in-the-loop is also called HITL.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



























