Human-in-the-loop (HITL) keeps a person involved where an agent should not act or be judged entirely automatically. Two distinct uses are approving a sensitive tool call before it runs and evaluating an agent’s output after it is produced. Approval controls execution; evaluation assesses quality. section 5.5.3 section 10.3
For tool approval, place the decision at the execution boundary. A before-tool callback can inspect the tool name and arguments. Safe calls proceed immediately; a call such as delete_file prompts the user. In the synchronous implementation, returning None allows execution, while returning a rejection string skips the tool and supplies that string as its result. The agent can then see the rejection and try another approach. section 5.5.3
Listing 5.27 Implementing approval callback
# List of dangerous tools requiring approval
DANGEROUS_TOOLS = ["delete_file", "send_email", "execute_sql"]
def approval_callback(context: ExecutionContext, tool_call: ToolCall):
"""Requests user approval before executing dangerous tools."""
# Execute immediately if not a dangerous tool
if tool_call.name not in DANGEROUS_TOOLS:
return None
print(f"\n
Dangerous tool execution requested"")
print(f"Tool: {tool_call.name}")
print(f"Arguments: {tool_call.arguments}")
response = input("Do you want to execute? (y/n): ").lower().strip()
if response == 'y':
print("
Approved. Executing...\n")
return None # Proceed with actual tool execution
else:
print("
Denied. Skipping execution.\n")
return f"User denied execution of {tool_call.name}"
The callback makes the approval boundary explicit: it checks a dangerous-tools list, shows the proposed arguments, and returns either permission to execute or a rejection result. section 5.5.3
A terminal prompt is useful for local development but blocks a production service while it waits. An asynchronous workflow instead pauses the agent, saves its execution state in a session, and returns a pending_confirmation result. A user can decide later through another interface; the agent restores the state and resumes. section 6.3 section 6.3.5
The pause-and-resume path needs three pieces of data:
- A PendingToolCall retains the original call and a confirmation message for the user. section 6.3.5
- A ToolConfirmation ties a decision to the tool-call ID, records approval or rejection, and can carry modified arguments. section 6.3.5
- An AgentResult distinguishes complete, pending_confirmation, and error states; pending calls can be serialized into session state and restored when execution resumes. section 6.3.5
Approve a file deletion
Configure delete_file with requires_confirmation=True. A request to delete temp.txt returns pending_confirmation and displays the proposed call. Create a ToolConfirmation with its tool-call ID and approved=True, then call run() again with the same session_id. The approved tool executes and the agent returns a completed response. Unlike the earlier dummy deletion tool, this example’s tool calls os.remove, making the approval decision consequential. section 5.5.3 section 6.3.8
Human evaluation serves a different checkpoint: people grade outputs with an understanding of context and nuance that simple reference-based metrics may miss. It is slow, expensive, and difficult to scale across frequent agent changes, so it can be combined with code-based evaluators, LLM judges, and end-user feedback rather than used for every case. section 10.3
