liveBook · Manning

Human-in-the-loop

Also called HITL

Human-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.

How 28 Manning books explain it

Introduction to Generative AI, Second Edition

Human-in-the-loop is a system-design pattern in which people participate in monitoring, evaluating, correcting, or governing an AI system.

How it is used
Places people in review, correction, and approval roles as AI systems assist with work or take actions.
How deep it goes
Explains review and approval patterns with concrete applications, while leaving implementation details unspecified.
Examples
Illustrates agent approval with Amazon Bedrock Agents and response comparison with human evaluations of Llama 2-Chat.
What you can do after
Evaluate oversight choices by considering task risk, reviewer expertise, and the limits of human judgment.

Read how Introduction to Generative AI, Second Edition explains human-in-the-loop →

AI Governance

An approach in which a person participates in an AI system’s operation, such as by reviewing or authorizing its outputs or actions.

How it is used
Provides review and approval checkpoints for sensitive AI outputs and actions, limiting unreviewed autonomy in production workflows.
How deep it goes
Details review triggers, approval thresholds, reviewer authority and supporting controls, while recognizing that oversight of rapid black-box agents remains unsolved.
Examples
Uses a fabricated clinical note and an insurance claim workflow to illustrate rubber-stamping and targeted review.
What you can do after
Operate/configure review workflows by selecting approval gates, giving reviewers authority and using override patterns to identify system weaknesses.

Read how AI Governance explains human-in-the-loop →

The Art of AI Product Development

Human evaluation is the assessment of model outputs by people using qualitative or task-specific judgments.

How it is used
Uses human judgment to evaluate model outputs and oversee AI decisions or agent actions where reliability, risk, or user needs warrant it.
How deep it goes
Explains oversight levels, review triggers, evaluation trade-offs, and safeguards without detailing a complete implementation.
Examples
Uses hybrid language-model evaluation, human review of hiring recommendations, and agent requests for missing interview data.
What you can do after
Apply human review to model evaluation and AI workflows by selecting oversight points and weighing review effort against risk.

Read how The Art of AI Product Development explains human-in-the-loop →

AI Agents in Action, Second Edition

Human evaluation is the assessment of system outputs or behavior by people applying judgment that may depend on context, policy, tone, or edge cases.

How it is used
Places human judgment in agent approval, escalation, output evaluation, and feedback loops.
How deep it goes
Details checkpoint triggers, reviewer context, waiting state, and failure safeguards; outlines rubric design and validation against human judgment.
Examples
Uses approval for sending email or transferring funds and a support-agent escalation tool for upset customers and large refunds.
What you can do after
Apply risk-based approval checkpoints and human review alongside automated checks and other safeguards.

Read how AI Agents in Action, Second Edition explains human-in-the-loop →

Architecting for Autonomy

A supervisory autonomy pattern in which a human monitors autonomous decisions and intervenes when behavior exceeds expected boundaries.

How it is used
Places human authority at bounded decision surfaces and distinguishes embedded review from supervisory oversight.
How deep it goes
Details trigger classes, handover requirements, validation, reviewer capacity, and risk-based selection between oversight postures.
Examples
Uses secured lending applications with low confidence, excessive exposure, or inconsistent documentation to illustrate human review.
What you can do after
Apply risk-based oversight by defining escalation triggers, testing handovers, and assessing reviewer capacity.

Read how Architecting for Autonomy explains human-in-the-loop →

LLM Evaluation and Alignment, The Foundational Ideas

A human preference signal is an evaluation signal derived from people's comparative or scalar judgments of output quality.

How it is used
Uses human quality judgments to evaluate open-ended outputs and oversee automated decisions.
How deep it goes
Details annotation guidance, sampling, golden-set monitoring, agreement metrics, and escalation procedures.
Examples
Uses a customer service chatbot response and crowd-labeling instructions to illustrate quality judgments.
What you can do after
Evaluate automated judges using human audit samples, agreement metrics, and a golden dataset.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains human-in-the-loop →

Generative AI in Action

Human evaluation is the assessment of an artifact or system by people using judgments, observations, surveys, or interviews.

How it is used
Adds human judgment and intervention to AI evaluation, content review, and critical decisions.
How deep it goes
Outlines evaluation criteria, evaluator selection, annotation, agreement analysis, and iterative improvement, alongside production oversight considerations.
Examples
Uses generated-code review, surveys, interviews, and crowdsourced annotation to illustrate human assessment.
What you can do after
Evaluate and understand AI outputs using human criteria, evaluator guidelines, and interrater agreement.

Read how Generative AI in Action explains human-in-the-loop →

Starting Data Analytics with Generative AI and Python

Oversight is the monitoring and supervision of an AI system's performance, impacts, and compliance by responsible people or groups.

How it is used
Supports human validation of automated analytics and oversight of AI performance and impacts.
How deep it goes
Outlines periodic expert review, automated issue flagging, and corrective responsibility without specifying a detailed review protocol.
Examples
Uses a Portuguese customer-review classifier trained on financial news and an AI-use policy with monitoring and reporting provisions.
What you can do after
Practice periodic review of data quality and AI outcomes alongside automated checks and error monitoring.

Read how Starting Data Analytics with Generative AI and Python explains human-in-the-loop →

Machine Learning System Design

Judgments or preference signals supplied by people about the quality or desirability of system outputs.

How it is used
Uses human judgments to evaluate outputs before deployment and weigh model improvements against regressions at release.
How deep it goes
Compares rating and preference judgments, discusses cost and throughput, and identifies review responsibilities and safeguards for automatic updates.
Examples
Uses search-result ratings and comparisons, image-search reasonableness judgments, and a release with three corrected errors and two new ones.
What you can do after
Evaluate whether human ratings or comparisons suit a quality or release decision, accounting for review costs and regressions.

Read how Machine Learning System Design explains human-in-the-loop →

Build a Multi-Agent System (from Scratch)

Human-in-the-loop is a design pattern in which people participate in or oversee stages of an automated process.

How it is used
Adds human input and oversight to agent tasks involving ambiguity, costly results, or risks of cascading errors.
How deep it goes
Details tool-based input, result approval, and stepwise supervision, including control tradeoffs and multi-agent extension requirements.
Examples
Uses restaurant selection to illustrate clarification and a critical bug-fix plan to illustrate approval and supervision.
What you can do after
Builds HITL support from scratch using a human-input tool, a result-approval gate, and a supervised task handler.

Read how Build a Multi-Agent System (from Scratch) explains human-in-the-loop →

Build an AI Agent (From Scratch)

Human evaluation is the direct assessment of system outputs by professional graders or domain experts.

How it is used
Uses human approval to gate sensitive agent actions and human evaluation to assess agent outputs.
How deep it goes
Implements synchronous callbacks and asynchronous pause-and-resume approval, while comparing human output evaluation with automated approaches.
Examples
Uses file deletion to demonstrate approval, rejection, and later resumption with a recorded decision.
What you can do after
Builds an approval workflow from callbacks, session state, pending tool calls, and confirmation decisions.

Read how Build an AI Agent (From Scratch) explains human-in-the-loop →

Designing AI Systems

Human review is the inspection and validation of an item by a person before it is accepted for an intended purpose.

How it is used
Uses human judgment to gate high-stakes tool actions and calibrate AI response evaluation.
How deep it goes
Details approval and timeout flows, policy configuration, annotation routing, and reviewer scoring.
Examples
Illustrates physician approval of diagnoses, supervisor review of large refunds, and rubric-based review of patient-intake traces.
What you can do after
Operate/configure approval gates and annotation queues with role-based routing, timeouts, sampling, and review rubrics.

Read how Designing AI Systems explains human-in-the-loop →

AI-Powered Search

A judgment is a human assessment of the relevance of a search result to a query or information need.

How it is used
Refines model-generated guesses into human-labeled training data and supplies relevance judgments for ranking.
How deep it goes
Defines a correction workflow and five label states, but provides no code that can generate the golden set automatically.
Examples
Uses question-answering model guesses in a CSV and movie-document relevance labels for example queries.
What you can do after
Practice manual labeling by classifying guesses, correcting answers, and refining a training set.

Read how AI-Powered Search explains human-in-the-loop →

AI Agents and Applications

How it is used
Routes uncertain or consequential agent requests to people before responses or actions are finalized.
How deep it goes
Outlines escalation triggers and feedback steps without implementing an approval mechanism.
Examples
Uses travel itineraries, disruption queries, accessibility requests, and high-value transactions as review cases.
What you can do after
Evaluate and understand which agent cases merit human approval and how review decisions can inform improvements.

Read how AI Agents and Applications explains human-in-the-loop →

"Looks Good to Me"

The human capacity to evaluate software decisions using nuance, context, history, and broader situational understanding.

How it is used
Reserves code-review attention for contextual decisions while automation handles repetitive checks.
How deep it goes
Explains contextual decisions through upgrade and variable-scope examples, but stops short of prescribing a formal judgment framework.
Examples
Uses library upgrades, removed service features, variable declarations, and AI-assisted pull-request reviews.
What you can do after
Evaluate software decisions, validate AI feedback, and make final code-review decisions using context and nuance.

Read how "Looks Good to Me" explains human-in-the-loop →

Digital Twins in Action

Human oversight is the supervision and authorization of an autonomous system's decisions and actions by people.

How it is used
Governs digital twin decisions and production changes when physical actions or their consequences warrant human approval.
How deep it goes
Defines authority tiers and approval boundaries, but stops short of implementing a human approval workflow.
Examples
Contrasts lighting, maintenance scheduling, and reactor pressure decisions, and illustrates an irrigation constraint with a policy rule.
What you can do after
Evaluate/understand when human approval is appropriate and distinguish it from constraints on automated actions.

Read how Digital Twins in Action explains human-in-the-loop →

Vibe Engineering

The monitoring and direction of an automated system by people who retain authority to identify errors and correct its course.

How it is used
Keeps agentic IDE work aligned with intent by having people approve, review, and correct agent actions.
How deep it goes
Explains selective approval and supervision costs, then contrasts human feedback with automated observability for unattended agents.
Examples
Uses file approvals, command validation, and the trade-off between thirty minutes saved and twenty minutes of supervision.
What you can do after
Operate agentic IDEs by calibrating approval policies to task risk and supervising novel or destructive actions.

Read how Vibe Engineering explains human-in-the-loop →

Data-Driven Cybersecurity

Human oversight is the review, supervision, or intervention by people in an automated system's operation and decisions.

How it is used
Positions human oversight as a safeguard for AI-driven security decisions and a governance check on cybersecurity performance.
How deep it goes
Explains human validation and regular assessment as oversight practices, without prescribing a detailed approval workflow.
Examples
Illustrates mistaken DDoS alerts from traffic surges and potentially disruptive access revocation based on behavioral data.
What you can do after
Practice/perform review of AI findings while assessing whether cybersecurity measures remain effective and aligned with organizational objectives.

Read how Data-Driven Cybersecurity explains human-in-the-loop →

Context Engineering

Human oversight is the involvement of people in monitoring, reviewing, or controlling consequential AI decisions and actions.

How it is used
Provides structured human review and approval for consequential or uncertain AI actions.
How deep it goes
Details risk-based routing, review, trace recording, evaluation feedback, and operational limits, alongside a basic interactive loop.
Examples
Uses customer-support escalation, a collaborative story-writing loop, and approval before database migrations or financial transactions.
What you can do after
Apply HITL by routing proposed actions to reviewers, recording decisions, and resuming workflows after approval.

Read how Context Engineering explains human-in-the-loop →

Financial AI in Practice

Human oversight is the involvement of people in reviewing, approving, or supervising system decisions and operation.

How it is used
Places human approval at consequential decision points in financial AI workflows.
How deep it goes
Explains approval checkpoints and their organizational rationale, without specifying an implementation procedure.
Examples
Illustrates oversight through post moderation, financial-advice approval, and credit-model governance review.
What you can do after
Evaluate where human approval and verification fit into financial AI workflows.

Read how Financial AI in Practice explains human-in-the-loop →

Effective Conversational AI

Human review is the examination of system outputs by people who assess their accuracy and contextual appropriateness.

How it is used
Controls conversational AI risk by keeping people responsible for reviewing or approving LLM outputs.
How deep it goes
Distinguishes periodic retroactive sampling from beforehand approval and combines automated scoring with human verification.
Examples
Uses weekly response sampling, copy editing, dialogue rewriting, and LLM-assisted evaluation of grounded and contextually relevant answers.
What you can do after
Operate conversational AI with human approval, monitor failures, and verify response accuracy and contextual relevance.

Read how Effective Conversational AI explains human-in-the-loop →

Understanding AI

Human oversight is the involvement of people in monitoring, evaluating, and controlling an AI system's outputs or behavior.

How it is used
Places human judgment between AI-generated outputs and consequential action, especially when models encounter external instructions.
How deep it goes
Provides a practical checklist for reviewing outputs but does not specify a detailed evaluation framework or technical control mechanism.
Examples
Uses external documents, websites, emails, trusted sources, output review, and high-stakes decisions as oversight scenarios.
What you can do after
Evaluate AI-generated outputs critically before using them, particularly when decisions carry significant consequences.

Read how Understanding AI explains human-in-the-loop →

First-Party Data Activation

A human-in-the-loop model is an AI system or workflow in which people participate in reviewing, approving, or directing model outputs.

How it is used
Uses human review and approval to manage AI-generated marketing outputs before they enter production.
How deep it goes
Recommends review or sampling by output type, but defers detailed roles and workflows.
Examples
Contrasts campaign-level targeting and assets with one-to-one personalized content that may require sampling.
What you can do after
Apply it by establishing review and approval processes, including sampling where personalized outputs are difficult to review individually.

Read how First-Party Data Activation explains human-in-the-loop →

Building Specialized AI Systems

Human-in-the-loop is a system design approach that incorporates human review or intervention into an automated data or model workflow.

How it is used
Uses human verification to improve synthetic labels in an automated training-dataset workflow.
How deep it goes
Details automated generation and subsequent verification, but does not specify reviewer criteria or approval procedures.
Examples
Uses analyst reports, generated summaries, prompt templates, and a DataFrame summary column as concrete workflow artifacts.
What you can do after
Apply it via automated summary generation followed by human verification before using outputs for training.

Read how Building Specialized AI Systems explains human-in-the-loop →

Building Reliable AI Systems

Human supervision of an AI system's outputs, decisions, and operation.

How it is used
Enforces human validation where AI errors could have serious professional consequences or complex logic requires careful review.
How deep it goes
Specifies legal review safeguards and agent approval steps, but gives no detailed human-review workflow implementation.
Examples
Uses fabricated legal citations, complex code logic, and agent execution checkpoints as review cases.
What you can do after
Apply human review before legal outputs reach court and retain careful review for complex code logic.

Read how Building Reliable AI Systems explains human-in-the-loop →

Grokking Statistics

Human in the loop is a system pattern in which people participate in reviewing, labeling, guiding, or validating automated processing.

How it is used
Frames human review as a risk-sensitive safeguard for automated predictions and as labor supporting model development.
How deep it goes
Explains the pattern through industry-risk comparisons and labeling workflows, but stops short of implementation procedures.
Examples
Contrasts Netflix recommendations with defense targeting and describes annotation by workers, doctors, and lawyers.
What you can do after
Operate predictive systems by planning human review, collecting new data, and updating and evaluating models as environments change.

Read how Grokking Statistics explains human-in-the-loop →

AI Model Evaluation

Human review is the assessment of model outputs or decisions by human evaluators.

How it is used
Uses human ratings to validate LLM judges and human review to support high-impact evaluation decisions.
How deep it goes
Identifies validation and review roles but does not detail a human-review workflow.
What you can do after
Evaluate judge reliability by comparing its outputs with human ratings.

Read how AI Model Evaluation explains human-in-the-loop →

Designing AI Agents

An approach in which a person intervenes in an automated process by supplying information, editing output, or judging a result.

How it is used
Uses people to supply information, edit output, or judge results within automated processes.
How deep it goes
Distinguishes broad human intervention from pre-execution approval, without detailing implementation of HITL itself.
Examples
Lists supplying missing information, editing a draft, and judging a result as human interventions.

Read how Designing AI Agents explains human-in-the-loop →

Also covered in

What to know first

Related concepts

Questions about human-in-the-loop

What is human-in-the-loop?

Human-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.

Which Manning books explain human-in-the-loop?

Introduction to Generative AI, Second Edition, AI Governance, The Art of AI Product Development, AI Agents in Action, Second Edition, Architecting for Autonomy, LLM Evaluation and Alignment, The Foundational Ideas, Generative AI in Action, Starting Data Analytics with Generative AI and Python and 20 more.

What should I know before learning human-in-the-loop?

Manning books that teach human-in-the-loop usually introduce language model, fine-tuning, AI agent and contract first.

What is the difference between human-in-the-loop and LLM-as-a-judge?

Human-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process. LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.

What is human-in-the-loop also called?

Human-in-the-loop is also called HITL.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.