liveBook · Manning

Human-in-the-loop

As explained in Build a Multi-Agent System (from Scratch)

Human-in-the-loop is a design pattern in which people participate in or oversee stages of an automated process.

Human-in-the-loop (HITL) puts a person into an agent’s task execution to provide input or oversight. It is useful when an agent cannot resolve ambiguity on its own, when an incorrect result is costly, or when an early error could cascade into later steps. section 1.4 section 8.0

Figure 1.9 An LLM agent processing loop with access to human operators. The processing loop is effectively paused each time a human operator is required to provide input.

The agent’s processing loop pauses when it needs input from a human operator. section 1.4

Choose an intervention point based on the task:

  • Tool-based HITL: Equip the agent with HumanInputTool. The agent decides when to call it, such as when it needs a person’s cuisine preference to narrow restaurant options. section 8.0
  • Result-approval HITL: Add an approval gate after execution. The operator accepts the result or rejects it with written feedback, which returns to the processing loop for revision. section 8.0 section 8.1
  • Supervised-execution HITL: Use SupervisedTaskHandler for stepwise execution. The operator can inspect and correct a step’s result before triggering the next step. section 8.0 section 8.1

Reviewing a critical bug-fix plan

An agent proposes a plan for shipping a critical fix. A human developer reviews it, accepts it or rejects it with feedback, and lets the agent revise a rejected plan. If errors during execution could have serious downstream effects, the developer can also supervise individual steps. section 8.1

Tool-based input is invoked at the agent’s discretion; approval and supervision are at the operator’s discretion. Approval is lighter weight than stepwise supervision and fits tasks where an occasional wrong result can be caught at the end. Supervision provides more visibility and can help develop or debug an agent. section 8.1

In a multi-agent system, agents can use HumanInputTool, but a single-terminal, single-process implementation needs locking and synchronization for safe requests. Approval can apply to individual tasks or the final aggregated result, depending on where oversight is needed. Simultaneously supervising every agent’s loop is impractical; in a hierarchy, a human can instead oversee the coordinator’s execution. section 8.5

How Build a Multi-Agent System (from Scratch) covers it

How deep it goes
Details tool-based input, result approval, and stepwise supervision, including control tradeoffs and multi-agent extension requirements.
Examples
Uses restaurant selection to illustrate clarification and a critical bug-fix plan to illustrate approval and supervision.
What you can do after
Builds HITL support from scratch using a human-input tool, a result-approval gate, and a supervised task handler.
How it connects
Connects agent processing loops, tool calls, result approval, debugging, and multi-agent systems.

Also explained in

Compare how 28 books explain human-in-the-loop →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Multi-Agent System (from Scratch). The section links point to the parts of the book it draws on.