Chapter 8: Human In the Loop
This introduction explains when human input, result approval, and stepwise supervision can help with agent tasks that should not be fully autonomous.
After this chapter you will be able to
- Input for Ambiguous Tasks. Explain when human input helps an agent make progress on an ambiguous task.
- Approval for Costly Results. Explain why human approval of task results is useful when errors would be costly.
- Stepwise Execution Supervision. Explain when seeing each step of an agent's execution helps a human oversee its work.
- Ambiguity and human input. Explain how a human-input tool can unblock an ambiguous task and who decides when to use it.
- High-stakes result approval. Describe how result approval helps with high-stakes tasks and how it is arranged.
- Supervised execution tradeoff. Explain how supervised execution differs from result approval and when its extra oversight is useful.
- Agent initiated human input. Explain how an agent uses HumanInputTool to resolve ambiguity during a task without changing the processing loop.
- Human input tool interface. Describe the HumanInputTool interface and how its prompt and optional choices constrain human input.
- Validated human input execution. Trace how HumanInputTool validates a call and obtains a human response through its helper.
- Result approval review flow. Trace how an agent’s task result is reviewed and what happens when the human approves or rejects it.
- Result approval execution flow. Trace how LLMAgent and TaskHandler obtain and handle a human review of a TaskResult.
- Rejected result step handling. Explain how get_next_step() handles a RejectedTaskResult differently from a TaskStepResult.
- Human control of execution. Explain how supervised execution differs from autonomous processing and what the human operator can do between steps.
- Supervised task handler design. Describe how run_supervised() and SupervisedTaskHandler support supervised execution, including the handler’s role in closing a task.
- Supervised result approval. Explain how a human operator accepts or rejects a proposed result during supervised execution.
- Background task restriction. Explain why and how SupervisedTaskHandler prevents use in an automatic processing loop.
- Supervised execution entry point. Explain how run_supervised() differs from run() and what it prepares and returns.
- Human controlled step cycle. Trace how the human advances a supervised task, including inspecting or modifying steps and inspecting their results.
- Final result disposition. Describe how to accept or reject a supervised task's final result and what to do if the next item is still a step.
- Inspector Setup and Launch. Describe how to prepare and launch the LLM Agent Inspector for an agent, including the role of the builder and optional default task.
- Stepwise Supervised Execution. Trace how a user starts and advances a supervised task in the Inspector, including the configuration available before execution.
- Result Review and Approval. Describe how to review and approve a supervised task result in the Inspector, and what can be done afterward.
- Tool-Based HITL in MAS. Explain how agents in a multi-agent system can seek human input and why the current HumanInputTool needs changes.
- Multi-Agent Result Approval. Explain the choice between approving individual agent tasks and approving the final aggregated result, including what influences the choice.
- Hierarchical Execution Supervision. Explain why supervising every agent's execution is impractical and how supervision can instead work in a hierarchical MAS.
What it builds on
- Designing a processing loop (chapter 4) · The agent's processing loop and its use of tools.
- Defining BaseTool (chapter 2) · BaseTool's required properties and call method.
- Implementing SimpleFunctionTool (chapter 2) · Validation of tool-call arguments against a tool's schema.
- Implementing LLMAgent: Part 2 (chapter 4) · The existing TaskHandler processing loop
- Wiring memory into task execution (chapter 7) · Recording task execution in memory
- Implementing TaskStep, TaskStepResult, and NextStepDecision (chapter 4) · TaskStep and TaskStepResult in the processing loop
- Implementing LLMAgentBuilder (chapter 5) · Using LLMAgentBuilder to specify an agent.
- Where and when to discover skills (chapter 6) · Skills are discovered at the start of task execution.
Section by section
Introduction
This introduction explains when human input, result approval, and stepwise supervision can help with agent tasks that should not be fully autonomous.
- Input for Ambiguous Tasks: Explain when human input helps an agent make progress on an ambiguous task.
Why it matters: A human can supply missing information when an agent cannot resolve an ambiguity on its own. - Approval for Costly Results: Explain why human approval of task results is useful when errors would be costly.
Why it matters: Reviewing consequential results can put a person in a position to catch costly mistakes. - Stepwise Execution Supervision: Explain when seeing each step of an agent's execution helps a human oversee its work.
Why it matters: Observing the execution step by step helps locate problems that a final result alone may not reveal.
Key terms: tool-based HITL, human input, result-approval HITL, human approval gate, task results, supervised HITL, stepwise task execution, processing loop
8.1 When—and How—should humans be in the loop?
This section explains when human involvement can resolve ambiguity, reduce high-stakes errors, or address security and compliance concerns in agent execution.
- Ambiguity and human input: Explain how a human-input tool can unblock an ambiguous task and who decides when to use it.
Why it matters: Human input can let an agent continue without guessing a missing preference. - High-stakes result approval: Describe how result approval helps with high-stakes tasks and how it is arranged.
Why it matters: Reviewing a consequential result can catch costly errors before the result is accepted. - Supervised execution tradeoff: Explain how supervised execution differs from result approval and when its extra oversight is useful.
Why it matters: Stepwise oversight can stop an early error from affecting later work. - Security and compliance oversight: Describe security and compliance reasons for requiring human oversight beyond ambiguity and high-stakes result quality.
Why it matters: Human oversight can enforce security boundaries and organizational requirements.
Key terms: ambiguous task execution, human-input tool, Tool-Based HITL, high-stakes task execution, result-approval step, Result-Approval HITL, supervised execution, Supervised-Execution HITL, result approval, security risks
8.2 Pattern 1: Tool-Based HITL
This section implements HumanInputTool so an agent can request human input during task execution without changing its processing loop.
- Agent initiated human input: Explain how an agent uses HumanInputTool to resolve ambiguity during a task without changing the processing loop.
Why it matters: This pattern adds human assistance using the agent's existing tool-calling behavior. - Human input tool interface: Describe the HumanInputTool interface and how its prompt and optional choices constrain human input.
Why it matters: The interface defines what the agent can ask and how it can restrict the human's answer. - Validated human input execution: Trace how HumanInputTool validates a call and obtains a human response through its helper.
Why it matters: Validation and a dedicated input helper connect the agent's tool call to a usable human response. - Human input interruption cases: State the two input interruption cases handled by the implementation.
Why it matters: Human input can be interrupted before an answer is available.
Key terms: Tool-Based HITL, HumanInputTool, processing loop, BaseTool, parameters_json_schema, prompt, choices, __call__(), validate_tool_call_arguments(), _prompt_human()
8.3 Pattern 2: Result-Approval HITL
This section explains how human review determines whether an agent’s task result is accepted or returned for further processing.
- Result approval review flow: Trace how an agent’s task result is reviewed and what happens when the human approves or rejects it.
Why it matters: This flow puts a human decision between producing a result and accepting it. - Review decision data structures: Distinguish the information held by ApprovalResult and RejectedTaskResult and describe which one returns to the processing loop.
Why it matters: These structures separate the review decision from the material needed to process a rejection.
Key terms: Result-Approval HITL, task result, human feedback, ApprovalResult, RejectedTaskResult
8.3.1 Updating TaskHandler and LLMAgent
This section updates LLMAgent and TaskHandler to review task results with a human and resume execution when a result is rejected.
- Result approval execution flow: Trace how LLMAgent and TaskHandler obtain and handle a human review of a TaskResult.
Why it matters: This flow makes human approval a gate on completing a task while allowing rejected work to be revised. - Rejected result step handling: Explain how get_next_step() handles a RejectedTaskResult differently from a TaskStepResult.
Why it matters: This conversion gives the next execution step the human feedback needed to revise the result. - Human review prompt helper: Describe how request_approval() delegates the operator interaction and how the synchronous helper is run.
Why it matters: The helper collects actionable feedback without keeping synchronous prompting on the event-loop thread. - Approval error fallback: Explain the fallback used when the approval interaction encounters an error without a human decision.
Why it matters: The fallback determines what happens when the system cannot obtain an actual review decision.
Key terms: with_approval, TaskResult, ApprovalResult, RejectedTaskResult, request_approval(), get_next_step(), TaskStepResult, TaskStep, run_step(), _prompt_for_approval()
8.4 Pattern 3: Supervised-Execution HITL
This section explains how supervised execution gives a human control over an agent’s processing loop and outlines the handler that supports it.
- Human control of execution: Explain how supervised execution differs from autonomous processing and what the human operator can do between steps.
Why it matters: This control lets an operator examine and adjust execution step by step when developing or debugging an agent. - Supervised task handler design: Describe how run_supervised() and SupervisedTaskHandler support supervised execution, including the handler’s role in closing a task.
Why it matters: The design reuses the existing step operations while giving supervised execution a way to finish the task.
Key terms: Supervised-Execution HITL, processing loop, get_next_step(), run_step(), LLMAgent.run_supervised(), SupervisedTaskHandler, TaskHandler, episodic memory
8.4.1 Implementing SupervisedTaskHandler
This section describes how SupervisedTaskHandler supports human review while reusing TaskHandler capabilities and preventing automatic execution.
- Supervised result approval: Explain how a human operator accepts or rejects a proposed result during supervised execution.
Why it matters: Supervision makes result approval possible as part of the human-controlled execution. - Inherited handler capabilities: Describe which existing TaskHandler capabilities SupervisedTaskHandler retains.
Why it matters: Supervision extends the existing handler rather than replacing its execution and memory capabilities. - Background task restriction: Explain why and how SupervisedTaskHandler prevents use in an automatic processing loop.
Why it matters: The restriction preserves the distinction between human-controlled execution and an automatic processing loop.
Key terms: SupervisedTaskHandler, TaskResult, complete(), reject(), RejectedTaskResult, Result-Approval HITL, TaskHandler, get_next_step(), run_step(), rollout
8.4.2 Updating LLMAgent
This section adds supervised execution to LLMAgent and demonstrates how a human advances, inspects, and finishes a task.
- Supervised execution entry point: Explain how run_supervised() differs from run() and what it prepares and returns.
Why it matters: The entry point gives the caller a handler for human-controlled execution rather than an ordinary task handler. - Human controlled step cycle: Trace how the human advances a supervised task, including inspecting or modifying steps and inspecting their results.
Why it matters: This cycle gives the human control over when each step runs and an opportunity to inspect it. - Final result disposition: Describe how to accept or reject a supervised task's final result and what to do if the next item is still a step.
Why it matters: The operator can either close the task with an acceptable result or continue processing after rejection.
Key terms: run_supervised(), run(), max_steps, with_approval, SupervisedTaskHandler, memories, get_next_step(), TaskStep, run_step(), TaskStepResult
8.4.3 Introducing the LLM Agent Inspector for supervised task executions
This section shows how to set up and use the LLM Agent Inspector to supervise and review an agent’s task execution through a browser UI.
- Inspector Setup and Launch: Describe how to prepare and launch the LLM Agent Inspector for an agent, including the role of the builder and optional default task.
Why it matters: The builder script connects your agent to the Inspector so you can test it interactively. - Stepwise Supervised Execution: Trace how a user starts and advances a supervised task in the Inspector, including the configuration available before execution.
Why it matters: Stepwise control lets you inspect and supervise the agent as it works. - Result Review and Approval: Describe how to review and approve a supervised task result in the Inspector, and what can be done afterward.
Why it matters: Reviewing the trajectory and approving the result makes the Inspector useful for iterative agent development.
Key terms: LLM Agent Inspector, agent-inspector, LLMAgentBuilder, agent_builder, default_task, Create session, get_next_step(), run_step(), skill discovery, scope
8.5 Extending to MAS: HITL
This section assesses how three human-in-the-loop patterns extend from individual LLM agents to multi-agent systems.
- Tool-Based HITL in MAS: Explain how agents in a multi-agent system can seek human input and why the current HumanInputTool needs changes.
Why it matters: This distinction shows why a tool that works for one agent needs adaptation when multiple agents can request input. - Multi-Agent Result Approval: Explain the choice between approving individual agent tasks and approving the final aggregated result, including what influences the choice.
Why it matters: The placement of approval determines whether human oversight can catch problems before they propagate. - Hierarchical Execution Supervision: Explain why supervising every agent's execution is impractical and how supervision can instead work in a hierarchical MAS.
Why it matters: A hierarchy preserves oversight without requiring one person to monitor every agent's processing loop at once.
Key terms: HumanInputTool, Tool-Based HITL, synchronization, locking, Result-Approval HITL, final aggregated task result, human oversight, Supervised-Execution HITL, hierarchical MAS, coordinator LLM agent
8.6 Summary
Where this is used later
- Supporting tool-based HITL in MAS (chapter 9) · Motivates supporting human input in multi-agent systems.
- Understanding the A2A elements used in a task lifecycle (chapter 10) · The motivation for requesting input when a task becomes ambiguous.
- Supporting subagents in our framework (chapter 9) · Supports the hierarchical use of subagents discussed as an alternative supervision structure.
Concepts explained in this chapter
- supervised-execution HITLSupervised-execution HITL is a human-in-the-loop pattern in which a human oversees, inspects, and can modify individual steps of an agent's task execution.
- result-approval hitlResult-Approval HITL is a human-in-the-loop pattern in which a human reviews and either approves or rejects an LLM agent's completed task result.
- tool-based HITLTool-based HITL is a human-in-the-loop pattern in which an agent invokes a tool to request information from a human during task execution.
- HumanInputToolHumanInputTool is a book-specific callable tool that obtains input from a human operator during agent execution.
- SupervisedTaskHandlerSupervisedTaskHandler is a book-specific TaskHandler variant whose task execution steps are manually supervised by a human operator.
- human-in-the-loopHuman-in-the-loop is a design pattern in which people participate in or oversee stages of an automated process.
- LLMAgentLLMAgent is the framework class that implements an LLM agent and its task-execution orchestration.
- get_next_step()get_next_step() is an asynchronous TaskHandler method that determines the next TaskStep or returns the final TaskResult.
- approval gateA human approval gate is a review mechanism that requires human approval of a task result and returns rejected results with feedback for revision.
- TaskHandlerTaskHandler is a framework helper class that orchestrates task execution across the processing loop and resolves to a TaskResult.
- run()run() is an LLMAgent method that accepts a user task, initiates its processing loop, and returns an awaitable that resolves to a TaskResult.
- LLM Agent InspectorLLM Agent Inspector is a book-specific developer tool that provides an interface for supervising LLM agent task execution.
- multi-agent systemA multi-agent system is a system composed of multiple interacting agents that cooperate, coordinate, or compete to achieve goals.
- TaskResultTaskResult is a framework data structure containing a task identifier and the result of executing that task.
- TaskStepTaskStep is a book-specific data structure representing one execution step associated with an overarching task.
- run_step()run_step() is an asynchronous TaskHandler method that executes one TaskStep and returns its TaskStepResult.
- processing loopA processing loop is a repeated execution structure in which an LLM agent plans, invokes tools, evaluates results, and continues through successive steps until a stopping condition is met.
- __call__() methodThe __call__() method is a Python special method that defines how an object behaves when invoked with function-call syntax.
- coordinator agentA coordinator LLM agent manages subagents or shared resources in a multi-agent system.
- subagentA subagent is an agent dispatched by another agent to work on a subtask contributing to an overall task.
- TaskStepResultTaskStepResult is a framework data structure containing the result of executing one agent sub-step.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
