liveBook · Manning

Supervised-execution HITL

As explained in Build a Multi-Agent System (from Scratch)

Supervised-execution HITL is a human-in-the-loop pattern in which a human oversees, inspects, and can modify individual steps of an agent's task execution.

A critical bug-fix agent may need oversight beyond approval of its final result: an operator can inspect and correct each step before errors cascade. Supervised-execution human-in-the-loop (HITL) gives the operator control over when the agent advances, making it useful for development and debugging as well as high-stakes tasks. section 8.1 section 8.4

Figure 8.10 Autonomous task execution advances the processing loop on its own, while supervised task execution gives a human operator full control over invocations of get_next_step() and run_step(), allowing them to pause and inspect or modify intermediate results as needed

Compare the automatic processing loop with the operator-controlled sequence of step requests and executions. section 8.4

Instead of automatically cycling between get_next_step() and run_step(), the operator invokes each method, inspects its output, and can modify it before continuing. This is more involved than result-approval HITL, which reviews the result after execution; unlike tool-based HITL, supervision is chosen by the operator rather than requested by the agent. section 8.1 section 8.4

A supervised execution follows this flow:

  • Call LLMAgent.run_supervised() to create a SupervisedTaskHandler. It loads memories before returning, and omits the automatic run’s max_steps and with_approval parameters because the operator controls execution. section 8.4.2
  • Call get_next_step(None) initially. Inspect the returned TaskStep, change it if needed, and pass it to run_step(). Inspect the resulting TaskStepResult before passing it to the next get_next_step() call. section 8.4.2
  • When get_next_step() returns a TaskResult instead of another TaskStep, accept it with complete(), or call reject() with feedback and pass the resulting RejectedTaskResult to get_next_step(). To stop instead, call abort(). section 8.4.2 section 8.4.1

SupervisedTaskHandler subclasses TaskHandler and reuses its step execution, rollout, and episodic-memory capabilities. complete() records the episode and sets the result; abort() records it and sets an exception. The handler cannot have a background asyncio.Task, because execution is triggered manually. section 8.4.1

Inspecting a Hailstone sequence

Give an agent a next_number tool and ask it to compute the sequence from 4 to 1. After run_supervised(), inspect the first TaskStep and run it. Continue with each TaskStepResult until get_next_step() returns a TaskResult, then review and complete it. One demonstrated result is 4, 2, 1. section 8.4.2

For a graphical workflow, LLM Agent Inspector drives the same supervised execution through a browser UI. A Python script supplies an LLMAgentBuilder as agent_builder and may provide a default_task; launch it with agent-inspector launch followed by the script path. section 8.4.3

Supervising every processing loop does not extend sensibly to a multi-agent system: one operator would have to oversee all agents simultaneously. In a hierarchical system, the operator can instead oversee the coordinator, which supervises subagent work. section 8.5

How Build a Multi-Agent System (from Scratch) covers it

How deep it goes
Walks through handler methods, step transitions, result rejection, memory recording, and Inspector setup, while omitting the skill-specific method walkthrough.
Examples
Demonstrates stepwise computation of the Hailstone sequence from 4 using a next_number tool.
What you can do after
Operate supervised execution through handler methods or the LLM Agent Inspector, inspecting steps and accepting, rejecting, or aborting a run.
How it connects
Connects human-in-the-loop execution with result approval, episodic memory, processing loops, and multi-agent systems.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Multi-Agent System (from Scratch). The section links point to the parts of the book it draws on.