Chapter 9: Governance: Constraints that enable freedom
The introduction explains why agents with consequential capabilities need governance and how clear boundaries can make delegation possible.
After this chapter you will be able to
- Capability and external consequences. Explain why governance becomes more important as an agent gains the ability to act outside the model, rather than merely suggest actions.
- Boundaries that enable delegation. Explain why an organization can give an agent more room to act when it sets clear boundaries while retaining responsibility for the effects.
- Bounded autonomy and drift. Explain how boundaries can give an agent room to act while keeping it aligned with the delegated objective, and why individually acceptable actions can add up to drift.
- Runtime authorization of proposals. Distinguish an agent's runtime choices from conventional software's more visible decision space, and explain why tool access alone does not authorize a particular action.
- Accountability through execution evidence. Describe what evidence is needed to account for an agent's consequential action and why an API log or natural-language chain-of-thought is insufficient.
- Containment of authorized mistakes. Explain why an authorized action still needs limits on its possible damage, using the coupon example.
- Autonomy levels and transitions. Explain how the book uses autonomy levels to discuss execution authority and human oversight, including how to frame a move from a current level to a next level.
- Hard ceilings and working room. Distinguish an agent's hard ceiling from the working room it receives for a particular task, and explain what can change that working room.
- Boundaries That Enable Autonomy. Explain why a clear, hard boundary can let an agent take more actions without repeatedly asking the user for permission.
- Pre-execution action routing. Explain that an Approval Gate identifies the particular action proposed and routes it to allow, deny, or review before execution. Distinguish its permission decision from broader human-in-the-loop interventions.
- Approval boundary calibration. Justify using an Approval Gate when actions differ in reversibility and impact, and describe what repeated unnecessary prompts or unseen consequential actions indicate about the allow boundary.
- Risk in familiar payments. Explain why a vendor-payment request can require review even when the supplier, amount, and credentials appear routine, and why reviewing every payment is not the answer.
- Deny allow ask routing. Describe what the deny, allow, and ask routes do and why the teaching gate applies them in that order.
- Impact and reversibility assessment. Explain why an action’s tool name does not determine its risk, and describe how impact and reversibility guide the default decision to proceed or seek review. State that explicit allow and deny rules take precedence over those defaults.
- Permission baselines and bypass risk. Explain that Claude Code’s six permission modes set baselines that may be further affected by project rules and runtime decisions, and why bypassPermissions is suitable only inside external isolation.
- PreToolUse decisions and limits. Describe how a PreToolUse hook can decide a tool call or leave normal permission flow in control, and explain why the hook alone is not a hard security boundary.
- Approval beyond the pause. Explain why a pre-tool-call gate or programmable pause is not the whole approval lifecycle, and identify the question that remains when execution resumes.
- Three-part approval record. Explain the distinct roles of intent, approval, and execution records in a production approval.
- Immutable durable intent. Describe what a durable intent preserves while awaiting review and why the reviewer approves that object rather than a reconstructable screen.
- Revalidation and execution controls. Trace what the gateway checks before resuming an approved action and distinguish the gaps closed by the execution controls.
- Gate-owned risk classification. Explain how the gate classifies a proposed action before applying policy rules, including why external effects are checked first and why production classification must verify caller-supplied claims.
- Deny and allow precedence. Trace how the teaching gate applies its separate deny and allow rules to a classified action, and explain how the remaining risk determines whether it proceeds or asks for approval.
- Approval fatigue and safe defaults. Explain why asking for approval on every file write can make review ineffective, and how to respond when approvals become automatic.
- Production policy and approval limits. Explain why tool-name rules alone are insufficient for an approval gate, what production policy and regression tests must check, and what approval still cannot guarantee.
- Limits beyond action approval. Explain why approval alone does not make an action safe and why Blast Radius Control asks how much one action can change.
- Containment boundaries and checks. Describe the limits this section says to set and how to check whether they contain an action.
- Nested blast radius boundaries. Explain why a sandbox alone cannot contain every harmful effect, and describe how capability restriction, execution isolation, and damage limitation work as nested boundaries.
- Task-scoped capability limits. Explain how least privilege limits the resources and operations an agent can reach, and distinguish what a sandbox restricts from what scoped credentials and API-side authorization restrict.
- Isolation boundary selection. Explain why the choice among hardened shared-kernel containers, OS-level sandboxes, user-space kernels, microVMs, and full VMs depends on the threat model, the effect of an escape, and operational costs.
- Sandbox reach boundaries. Describe how the sandboxed Bash example limits filesystem writes and outbound network access, and distinguish those limits from read permissions and business correctness.
- Limits on authorized actions. Explain why an action that passes capability and sandbox checks still needs limits, and describe how rate limits and budget caps prevent repeated actions from accumulating harm or cost.
- Intervention and recovery controls. Describe how time locks and Saga-style rollback limit damage in different ways, and explain why actions without clean compensation need tighter earlier ceilings.
- Hard blast radius boundary. Distinguish the hard blast-radius ceiling from the usable space an agent may gain as its trust level rises, using the section’s operational examples.
- Policy envelope and runtime boundary. Explain how SandboxConfig, individual checks, and execute divide responsibility, and why SandboxedExecutor is not itself an OS sandbox. Describe what the supplied runtime must enforce.
- Independent containment layers. Explain why gates, limiters, sandboxes, and kill switches do not provide defense in depth if they share a process or credential, and why higher-risk execution may need a separate process or microVM.
- Damage from permitted actions. Explain why Blast Radius Control cannot establish that an authorized action is correct, and describe the cheaper evidence needed before a consequential commitment.
- Two Timescales of Commitment. Explain how Progressive Commitment works within a run and across runs, and how this differs from judging only the action immediately in front of the agent.
- Evidence for Autonomy Changes. Explain what makes evidence for increasing or decreasing autonomy credible, and identify the boundaries and outcomes the section says to track.
- Staged verification before commitment. Trace how an agent checks a proposed high-risk action before committing it: freeze the proposal and resource version, perform cheaper checks and a dry run, then revalidate the same proposal before the live operation. Explain why the full chain is reserved for consequences that justify its latency.
- Evidence based trust promotion. Distinguish evidence that justifies one action within a session from accumulated evidence that can support greater autonomy across sessions. Describe the conditions and scope for production promotion, including owner approval and fresh evidence after a transition or model upgrade.
- Human sovereignty triggers. Explain the three conditions that require human involvement—critical operations, insufficient evidence, and ethical sensitivity—and why a high trust level does not remove that requirement.
- Trust Boundary Enforcement Points. Distinguish the enforcement boundaries of model-traffic filters, output policy checks, and runtime tool/API policies. Explain why a tool call that bypasses the AgentCore Gateway also bypasses its policy boundary, and why these mechanisms do not decide trust-level promotion.
- Configured permission baseline. Explain that Claude Code's permission mode is chosen by a user or administrator, not promoted by the agent, and distinguish the stated limits of dontAsk and bypassPermissions.
- Per-action hook checks. Describe how PreToolUse hooks check an individual action before its tool runs, and explain why verification and commitment remain ordered even when matching hooks run in parallel.
- Outcome evidence for trust. Explain how recording an action outcome updates both cumulative results for a later promotion decision and a consecutive-error count that detects a new failure streak.
- Asymmetric trust level changes. Describe why promotion requires all three example thresholds, whereas consecutive errors prompt a demotion attempt and reset the current evidence before promotion is checked.
- Trust-aware approval routing. Explain how trust level changes which actions the Approval Gate sends for human review without granting tool permission or removing sovereignty triggers.
- Within-run probe validity. Explain why a successful dry run may not predict a successful commit and how to keep the probe representative of the live action.
- Cross-run promotion evidence. Explain why a high success rate on easy work does not justify promotion and what the evidence must cover.
- Connected action evidence. Explain why an agent’s visible action is not enough for accountability and describe the record needed to reconstruct what happened after a run.
- Replayable agent trajectories. Explain why replaying a user prompt is insufficient and what must be preserved to examine an agent run later.
- Correlated governance evidence. Describe how the Observability Harness joins distributed signals into a causal record and distinguish execution records from evaluation and replay evidence.
- Evidence-driven governance changes. Explain how collected evidence leads from a bad outcome to a corrective change and a new governance decision.
- Observability platform selection criteria. Explain how to assess whether an observability platform can support a governed agent’s evidence path, and why a feature table, license label, or self-hosting claim is not enough.
- Metrics and trajectory evidence. Explain why investigation starts with task outcomes and cost, moves through process steps to model and API calls, and needs trajectories and source evidence to explain particular decisions.
- Business-observable task success. Describe where to measure task success and why downstream use matters even though it is an incomplete signal.
- Organizational observability boundary. Explain what Claude Code supplies at the observability integration boundary and what a regulated deployment must add.
- Protected model call evidence. Explain what an LLM-call record captures and how an authorized investigator can inspect its messages and output without putting their contents in general telemetry.
- Call results versus task outcomes. Distinguish model and tool-call results from the overall task outcome, including where approval overrides are preserved.
- Teaching layer and production export. Describe the teaching instrumentation layer's span lifecycle and dashboard, and state what is needed to export its data for production observability.
- Trace-based failure diagnosis. Trace an overcompensation incident from the coupon proposal through the evidence the model saw and the recorded policy decision. Distinguish a bad proposal from a gate that wrongly admitted it.
- Evidence-driven governance repair. Describe how to convert the incident into regression tests and use their results to guide a staged fix. Include both the excessive proposal and a valid within-limit request.
- Trace volume and representative sampling. Explain why capturing every span may still leave an agent poorly observed, and describe how alerting and stratified sampling make important paths easier to find without favoring easy work.
- Execution traces and reasoning limits. Distinguish a natural-language chain-of-thought from an execution record, and explain what observable traces can and cannot establish.
- Four-pattern governance pipeline. Describe how Approval Gate, Progressive Commitment, Blast Radius Control, and Observability Harness address different decisions about one proposed action.
- Two-stage action authorization. Explain why unusable tools are removed during assembly while the selected tool and its arguments still require a runtime check against trusted task state.
- Task-wide cap enforcement. Apply the compensation example to explain why a proposal can fail a task-wide cap despite being individually approvable, and how a production gate protects that cap across concurrent proposals.
- Risk-proportionate governance cost. Justify varying governance overhead according to an action's worst credible effect, using the section's examples.
- Governance controls and evidence. Explain that regulation asks not just what governance controls an agent has, but which obligations those controls satisfy and what evidence the organization can produce when challenged.
- Runtime Policy and Evidence. Explain why no single governance pattern proves compliance and describe the distinct evidence the four patterns contribute to a governed system.
- Argus governance boundary. Explain which Argus operations currently pass through the governance facade and why a production build must extend that boundary.
- Evidence-backed human review. Describe what the host preserves and presents when a governed tool call requires human approval.
- Provisional ASK routing. Trace how the facade handles an ASK decision, an explicit DENY, and the tool call that proceeds.
- Governance as allocated trust. Explain how the four governance patterns allocate trust and how governance differs from perception at the model boundary.
What it builds on
- Risk follows the effect (chapter 6) · Connects action risk to an action's effects.
- Set trust boundaries before granting autonomy (chapter 2) · Introduces setting trust boundaries before granting autonomy.
- Eligibility comes before selection (chapter 6) · Introduces checking tool eligibility before selection.
- Pattern: Plan-and-Execute (chapter 6) · The earlier plan-and-execute discussion covers recovery in multi-step workflows.
- Reveal capability in layers (chapter 6) · Earlier discussion of revealing a smaller set of tool capabilities for selection.
- MCP and A2A are the plumbing (chapter 8) · A2A as collaboration plumbing.
- (chapter intro) (chapter 3) · Perception as the agent's view of information.
Section by section
Introduction
The introduction explains why agents with consequential capabilities need governance and how clear boundaries can make delegation possible.
- Capability and external consequences: Explain why governance becomes more important as an agent gains the ability to act outside the model, rather than merely suggest actions.
Why it matters: The need for governance grows with what an agent can change, not just with how capable it seems. - Boundaries that enable delegation: Explain why an organization can give an agent more room to act when it sets clear boundaries while retaining responsibility for the effects.
Why it matters: Clear limits can support more useful autonomy rather than simply restricting an agent.
Key terms: governance, capability, blast radius, delegates choices, retains responsibility, clear boundary
9.1 What is governance? Autonomy without accountability is a liability
This section explains why agent governance must preserve useful autonomy while checking runtime proposals, their cumulative direction, and the consequences of mistakes.
- Bounded autonomy and drift: Explain how boundaries can give an agent room to act while keeping it aligned with the delegated objective, and why individually acceptable actions can add up to drift.
Why it matters: A sequence can undermine the task even when each step looks reasonable on its own. - Runtime authorization of proposals: Distinguish an agent's runtime choices from conventional software's more visible decision space, and explain why tool access alone does not authorize a particular action.
Why it matters: Permission to use a service is not permission to make every possible change through it. - Accountability through execution evidence: Describe what evidence is needed to account for an agent's consequential action and why an API log or natural-language chain-of-thought is insufficient.
Why it matters: An investigator needs to reconstruct what was proposed, allowed, and changed—not merely who called an endpoint. - Containment of authorized mistakes: Explain why an authorized action still needs limits on its possible damage, using the coupon example.
Why it matters: Passing an authorization check does not eliminate the cost of an agent's mistake.
Key terms: delegated objective, least agency, authorization, API key, role-based access control (RBAC), runtime, accountability, execution evidence, policy decision, containment
9.1.1 The governance spectrum
This section presents autonomy as a graded operating choice, bounded by firm limits and adjusted to the evidence of trust and the risk of the task.
- Autonomy levels and transitions: Explain how the book uses autonomy levels to discuss execution authority and human oversight, including how to frame a move from a current level to a next level.
Why it matters: A graded scale lets a team discuss a specific increase in authority instead of treating autonomy as all or nothing. - Hard ceilings and working room: Distinguish an agent's hard ceiling from the working room it receives for a particular task, and explain what can change that working room.
Why it matters: Even a proven agent may need tighter limits when a task is unfamiliar or hard to undo.
Key terms: autonomy spectrum, execution authority, human oversight, operating arrangements, hard ceiling, working room, trust earned from observed outcomes, risk of the current task
9.1.2 Constraints enable freedom
The section explains how firm operating boundaries can reduce routine interruptions while keeping an agent’s actions subject to review.
- Boundaries That Enable Autonomy: Explain why a clear, hard boundary can let an agent take more actions without repeatedly asking the user for permission.
Why it matters: Good constraints can make an agent both easier to govern and less burdensome to use. - Distinct Governance Controls: Distinguish the sandbox and approval controls in the Codex example, and describe what an Automated Reasoning check does—and does not decide—in the Bedrock example.
Why it matters: Separating limits, review decisions, and policy findings makes clear which control is responsible for each decision.
Key terms: hard boundary, sandboxing, permission prompts, sandbox, approval policy, Automated Reasoning checks, formal policy, findings
9.2 Pattern: Approval Gate
Introduces the Approval Gate as a pre-execution decision that routes proposed actions to allow, deny, or human review, and explains why those boundaries need calibration.
- Pre-execution action routing: Explain that an Approval Gate identifies the particular action proposed and routes it to allow, deny, or review before execution. Distinguish its permission decision from broader human-in-the-loop interventions.
Why it matters: This distinction keeps human review focused on actions that actually require it. - Approval boundary calibration: Justify using an Approval Gate when actions differ in reversibility and impact, and describe what repeated unnecessary prompts or unseen consequential actions indicate about the allow boundary.
Why it matters: A well-placed boundary preserves human attention for consequential decisions without letting them escape review.
Key terms: Approval Gate, human-in-the-loop (HITL), deny, allow, review, reversibility, impact, approval fatigue, allow boundary
9.2.1 Risk classification and the approval ladder
This section explains how an Approval Gate routes a payment proposal by risk while keeping the proposal stable enough for meaningful review before execution.
- Risk in familiar payments: Explain why a vendor-payment request can require review even when the supplier, amount, and credentials appear routine, and why reviewing every payment is not the answer.
Why it matters: Risk-based review protects consequential changes without overwhelming reviewers. - Deny allow ask routing: Describe what the deny, allow, and ask routes do and why the teaching gate applies them in that order.
Why it matters: The route determines which actions stop, proceed, or need human judgment. - Frozen proposal review stages: Distinguish routing a frozen proposal from reviewing and revalidating it before execution.
Why it matters: An approval is useful only if it applies to the proposal that is actually considered for execution.
Key terms: approval fatigue, changed account, deny rule, allow rule, ask route, deny → allow → ask, frozen proposal, reviewing, revalidating
9.2.2 Classifying action risk
This subsection classifies action risk by impact and reversibility rather than by tool name, while giving explicit allow and deny rules precedence over the resulting defaults.
- Impact and reversibility assessment: Explain why an action’s tool name does not determine its risk, and describe how impact and reversibility guide the default decision to proceed or seek review. State that explicit allow and deny rules take precedence over those defaults.
Why it matters: Judging the proposed effect rather than the tool helps apply the right level of oversight.
Key terms: impact, reversibility, action risk classification, explicit deny and allow rules
9.2.3 In production: Claude Code's permission modes
Compares production approval controls and shows why a permission mode, hook, sandbox, or pause point is only part of a governed approval process.
- Permission baselines and bypass risk: Explain that Claude Code’s six permission modes set baselines that may be further affected by project rules and runtime decisions, and why bypassPermissions is suitable only inside external isolation.
Why it matters: Treating bypass mode as ordinary permission policy can leave actions without the intended boundary. - PreToolUse decisions and limits: Describe how a PreToolUse hook can decide a tool call or leave normal permission flow in control, and explain why the hook alone is not a hard security boundary.
Why it matters: A convenient hook attachment point must not be mistaken for guaranteed enforcement. - Technical and review boundaries: Distinguish Codex’s sandbox enforcement from its approval policy, and describe what LangGraph’s interrupt() supplies versus what the application must provide.
Why it matters: A pause or review rule does not replace enforcement of what the agent can actually do. - Approval beyond the pause: Explain why a pre-tool-call gate or programmable pause is not the whole approval lifecycle, and identify the question that remains when execution resumes.
Why it matters: An approval is meaningful only if it still applies to the action that ultimately runs.
Key terms: permission modes, bypassPermissions, --dangerously-skip-permissions, external isolation, PreToolUse hooks, permissionDecision, normal permission flow, fail-closed gateway, sandbox layer, approval policy layer
9.2.4 Approval is a contract across time
This section explains how to keep a human approval tied to a specific intended action and its eventual execution, even when circumstances change during the wait.
- Three-part approval record: Explain the distinct roles of intent, approval, and execution records in a production approval.
Why it matters: Separate records let you compare the authorized action with the action that actually ran. - Immutable durable intent: Describe what a durable intent preserves while awaiting review and why the reviewer approves that object rather than a reconstructable screen.
Why it matters: A durable intent prevents the action presented for review from being silently reconstructed as a different action at execution time. - Revalidation and execution controls: Trace what the gateway checks before resuming an approved action and distinguish the gaps closed by the execution controls.
Why it matters: An approval that was valid when given is not sufficient if the action or its target changes before execution.
Key terms: intent, approval, execution, durable intent, digest, resume coordinate, approval lifetime, business preconditions, idempotency_key, one-time execution token
9.2.5 Building it
This section builds a teaching Approval Gate that classifies proposed actions, applies ordered deny and allow policies, and records why each action proceeds or requires approval.
- Gate-owned risk classification: Explain how the gate classifies a proposed action before applying policy rules, including why external effects are checked first and why production classification must verify caller-supplied claims.
Why it matters: The gate cannot enforce a reliable approval policy if an action can declare its own risk or hide a critical external effect. - Deny and allow precedence: Trace how the teaching gate applies its separate deny and allow rules to a classified action, and explain how the remaining risk determines whether it proceeds or asks for approval.
Why it matters: Rule precedence keeps a broad permission from silently overriding a denial or a critical external effect. - Recorded routing reasons: Describe what the teaching gate records about each decision and distinguish that routing reason from a fuller review packet.
Why it matters: A recorded reason makes the gate's decision inspectable without mistaking the teaching example for a complete approval workflow.
Key terms: risk_level, classify_risk, policy tier, CRITICAL, external effects, deny rules, allow rules, precedence, audit log, reason
9.2.6 When it breaks
This section explains how approval gates fail through reviewer fatigue and incomplete policy checks, and why approval cannot guarantee a safe result.
- Approval fatigue and safe defaults: Explain why asking for approval on every file write can make review ineffective, and how to respond when approvals become automatic.
Why it matters: An approval prompt protects little if reviewers have learned to accept it automatically. - Production policy and approval limits: Explain why tool-name rules alone are insufficient for an approval gate, what production policy and regression tests must check, and what approval still cannot guarantee.
Why it matters: A gate must check the effect an action can have, not just the tool name or the fact that someone approved it.
Key terms: approval fatigue, approval-to-rejection ratio, allow path, tool-name matching, arguments, resources, environment, regression suite, critical external effect
9.3 Pattern: Blast Radius Control
The section introduces Blast Radius Control as a way to limit the damage an action can cause even after it has been approved.
- Limits beyond action approval: Explain why approval alone does not make an action safe and why Blast Radius Control asks how much one action can change.
Why it matters: Permission to act and limits on the consequences of acting solve different problems. - Containment boundaries and checks: Describe the limits this section says to set and how to check whether they contain an action.
Why it matters: Explicit limits and failure tests show whether one bad action can escape its intended scope.
Key terms: Approval Gate, Blast Radius Control, task boundary, capabilities, execution boundary, quantitative limits, fault injection
9.3.1 Nested containment: Sandbox inside sandbox
This subsection explains why blast radius control needs nested boundaries beyond a sandbox.
- Nested blast radius boundaries: Explain why a sandbox alone cannot contain every harmful effect, and describe how capability restriction, execution isolation, and damage limitation work as nested boundaries.
Why it matters: A permitted tool can still cause harm, so controlling where code runs is only part of containing its effects.
Key terms: sandbox, blast radius control, capability restriction, execution isolation, damage limitation, Hierarchy topology
9.3.2 Layer 1: Capability restriction (least privilege)
This section explains how to give an agent only the capabilities its task requires and expand them only when evidence warrants it.
- Task-scoped capability limits: Explain how least privilege limits the resources and operations an agent can reach, and distinguish what a sandbox restricts from what scoped credentials and API-side authorization restrict.
Why it matters: A sandbox alone cannot enforce every boundary on what an agent is allowed to change. - Evidence-based capability expansion: Describe the section's default for setting and relaxing an agent's capability boundaries.
Why it matters: This approach avoids granting broader access merely for convenience while leaving a way to resolve genuine blockers.
Key terms: capability restriction, least privilege, sandbox, scoped credentials, API-side authorization, narrowest capability, boundary, real blocker
9.3.3 Layer 2: Execution isolation (sandboxing)
This section explains how execution isolation limits an agent’s technical reach and how to choose an isolation boundary based on threats and operational trade-offs.
- Isolation boundary selection: Explain why the choice among hardened shared-kernel containers, OS-level sandboxes, user-space kernels, microVMs, and full VMs depends on the threat model, the effect of an escape, and operational costs.
Why it matters: The right sandbox is the one whose boundary and costs fit the risk, not simply the strongest-sounding option. - Sandbox reach boundaries: Describe how the sandboxed Bash example limits filesystem writes and outbound network access, and distinguish those limits from read permissions and business correctness.
Why it matters: A sandbox can limit where an agent acts without ensuring that every permitted action is appropriate. - Sandbox performance claims: Distinguish E2B’s advertised end-to-end sandbox startup time from Firecracker’s component startup and footprint specifications.
Why it matters: Keeping component and service metrics separate avoids overstating the performance of an isolation choice.
Key terms: threat model, shared-kernel containers, user-space kernel, microVM, full VM-based isolation, filesystem isolation, read access, network isolation, proxy, technical reach
9.3.4 Layer 3: Damage limitation
This section explains how damage-limitation controls contain authorized actions and why increasing an agent’s trust must never raise its hard blast-radius ceiling.
- Limits on authorized actions: Explain why an action that passes capability and sandbox checks still needs limits, and describe how rate limits and budget caps prevent repeated actions from accumulating harm or cost.
Why it matters: A permitted action can become damaging when an agent repeats it or spends without a ceiling. - Intervention and recovery controls: Describe how time locks and Saga-style rollback limit damage in different ways, and explain why actions without clean compensation need tighter earlier ceilings.
Why it matters: Some mistakes can be caught before execution or compensated afterward, while others require stricter limits up front. - Hard blast radius boundary: Distinguish the hard blast-radius ceiling from the usable space an agent may gain as its trust level rises, using the section’s operational examples.
Why it matters: An agent can earn more autonomy without gaining the ability to cause an unbounded or forbidden impact.
Key terms: damage limitation, rate limits, budget caps, time lock, cancellation window, revalidation, Saga-style rollback, compensating actions, blast radius, hard ceiling
9.3.5 Building it
This section shows how to assemble blast-radius checks around an execution function while leaving physical isolation to an external runtime.
- Policy envelope and runtime boundary: Explain how SandboxConfig, individual checks, and execute divide responsibility, and why SandboxedExecutor is not itself an OS sandbox. Describe what the supplied runtime must enforce.
Why it matters: Policy checks cannot replace the runtime isolation that contains execution. - Path rate and cost checks: Describe the path, rate, and cumulative-cost checks, including what the rate window and estimated cost measure. Explain why production cost estimates need trusted data and reconciliation.
Why it matters: Precise, separately testable limits make the execution policy more dependable.
Key terms: SandboxConfig, SandboxedExecutor, execute_fn, policy envelope, isolated runtime, Layer 1, Layer 3, estimated_cost, cumulative_cost, max_cost_per_task_usd
9.3.6 When it breaks
This section explains two ways Blast Radius Control can fail: supposedly separate defenses can share a point of compromise, and permitted actions can still cause damage.
- Independent containment layers: Explain why gates, limiters, sandboxes, and kill switches do not provide defense in depth if they share a process or credential, and why higher-risk execution may need a separate process or microVM.
Why it matters: Several controls only reduce risk if one compromise cannot bypass all of them. - Damage from permitted actions: Explain why Blast Radius Control cannot establish that an authorized action is correct, and describe the cheaper evidence needed before a consequential commitment.
Why it matters: Limiting where and how an agent acts does not ensure that its permitted action has the right effect.
Key terms: Blast Radius Control, defense in depth, sandbox, kill switch, microVM, legitimate-action damage, limiter, Progressive Commitment, dry run, validation
9.4 Pattern: Progressive Commitment
Progressive Commitment adds a time dimension to governance: an agent earns consequential actions through reversible checks within a run, while its autonomy changes across runs based on observed outcomes.
- Two Timescales of Commitment: Explain how Progressive Commitment works within a run and across runs, and how this differs from judging only the action immediately in front of the agent.
Why it matters: It separates checking an action before taking it from changing how much freedom the agent has over time. - Evidence for Autonomy Changes: Explain what makes evidence for increasing or decreasing autonomy credible, and identify the boundaries and outcomes the section says to track.
Why it matters: More permission is only justified when external, representative evidence supports it and there is a way to take it back.
Key terms: Progressive Commitment, reversible checks, autonomy, harness, business outcome, promotion and demotion criteria, human overrides
9.4.1 Earning trust through evidence
Progressive Commitment uses checks before individual actions and accumulated operating evidence across sessions to grant bounded autonomy while retaining the ability to withdraw it.
- Staged verification before commitment: Trace how an agent checks a proposed high-risk action before committing it: freeze the proposal and resource version, perform cheaper checks and a dry run, then revalidate the same proposal before the live operation. Explain why the full chain is reserved for consequences that justify its latency.
Why it matters: A check is most useful when it can catch a mistake before the costly or consequential operation. - Evidence based trust promotion: Distinguish evidence that justifies one action within a session from accumulated evidence that can support greater autonomy across sessions. Describe the conditions and scope for production promotion, including owner approval and fresh evidence after a transition or model upgrade.
Why it matters: Separating permission for one action from lasting trust prevents a narrow success from becoming broad autonomy. - Incident driven trust withdrawal: Explain why the trust ladder must allow demotion as well as promotion, and describe the response to a critical, severe, or repeated incident.
Why it matters: A trust system that only promotes agents cannot respond promptly when new evidence shows they are unsafe.
Key terms: Progressive Commitment, staged verification, dry run, Chain topology, Blast Radius Control, trust ladder, staged trust escalation, production promotion, trust record, automatic demotion
9.4.2 Human sovereignty triggers
This section identifies when an agent must seek human input regardless of its earned trust level.
- Human sovereignty triggers: Explain the three conditions that require human involvement—critical operations, insufficient evidence, and ethical sensitivity—and why a high trust level does not remove that requirement.
Why it matters: The agent can handle more routine work as trust grows without taking over decisions reserved for humans.
Key terms: human-sovereignty triggers, critical operations, insufficient evidence, ethical sensitivity, should_ask_human, trust level
9.4.3 Enforcing the current trust boundary
This subsection compares where guardrail mechanisms enforce a trust-level policy and distinguishes enforcement from deciding whether an agent has earned more trust.
- Trust Boundary Enforcement Points: Distinguish the enforcement boundaries of model-traffic filters, output policy checks, and runtime tool/API policies. Explain why a tool call that bypasses the AgentCore Gateway also bypasses its policy boundary, and why these mechanisms do not decide trust-level promotion.
Why it matters: A guardrail cannot enforce a policy on calls outside its boundary, and applying a trust-level policy is not the same as granting more trust.
Key terms: enforcement point, model traffic, formal policies, runtime policies, AgentCore Gateway, trust level
9.4.4 A configured baseline and an earned trust level
This section distinguishes a human-configured permission baseline from per-action hooks that check a proposed tool call before it runs.
- Configured permission baseline: Explain that Claude Code's permission mode is chosen by a user or administrator, not promoted by the agent, and distinguish the stated limits of dontAsk and bypassPermissions.
Why it matters: The agent cannot grant itself broader authority simply because an action seems useful. - Per-action hook checks: Describe how PreToolUse hooks check an individual action before its tool runs, and explain why verification and commitment remain ordered even when matching hooks run in parallel.
Why it matters: A configured mode does not replace checks on the particular action about to run.
Key terms: permission modes, configured baseline, dontAsk, bypassPermissions, PreToolUse hooks, frozen proposal, commitment chain
9.4.5 Building it
This section shows how recorded outcomes adjust an agent’s cross-session trust level, how that level affects approval routing, and what the example implementation lacks for production use.
- Outcome evidence for trust: Explain how recording an action outcome updates both cumulative results for a later promotion decision and a consecutive-error count that detects a new failure streak.
Why it matters: Separate evidence for sustained success and recent failures lets trust respond differently to each. - Asymmetric trust level changes: Describe why promotion requires all three example thresholds, whereas consecutive errors prompt a demotion attempt and reset the current evidence before promotion is checked.
Why it matters: The ordering prevents a recent incident from being overridden by successes accumulated before it. - Trust-aware approval routing: Explain how trust level changes which actions the Approval Gate sends for human review without granting tool permission or removing sovereignty triggers.
Why it matters: Trust can reduce routine review without becoming a blanket authorization to act. - Production promotion safeguards: Distinguish the listings’ demonstration from a complete production promotion service, including the additional rules, approvals, and protected persistent state a deployment needs.
Why it matters: The example’s trust calculations alone are not enough to make promotion safe or durable in a deployment.
Key terms: TrustManager.record_action, cumulative results, consecutive-error count, TrustMetrics, promotion, demotion, Observe, EscalationThresholds, Approval Gate, should_ask_human
9.4.6 When it breaks
This section explains how progressive commitment can fail when a probe no longer matches the action being committed or when promotion evidence does not represent the work ahead.
- Within-run probe validity: Explain why a successful dry run may not predict a successful commit and how to keep the probe representative of the live action.
Why it matters: A passed probe is useful only if it still tests the action the agent will take. - Cross-run promotion evidence: Explain why a high success rate on easy work does not justify promotion and what the evidence must cover.
Why it matters: An impressive success rate can overstate readiness if it comes only from easy tasks.
Key terms: within-run clock, probe, dry run, commit, snapshot, cross-run clock, stratified sampling, task difficulty, operation classes, promotion
9.5 Pattern: Observability Harness
Introduces the Observability Harness as a way to preserve connected evidence of an agent’s decisions and results so its actions can be debugged, evaluated, and audited.
- Connected action evidence: Explain why an agent’s visible action is not enough for accountability and describe the record needed to reconstruct what happened after a run.
Why it matters: A business outcome alone cannot show why the agent acted or whether the external system changed. - Harness design and checks: Describe the design choices for an Observability Harness and the measurements suggested for checking its operation.
Why it matters: Choosing what evidence survives—and checking for gaps—makes the record useful without ignoring access, redaction, or overhead.
Key terms: Observability Harness, accountability, inputs, permissions, actions, outcomes, trace boundary, correlation identifiers, retention period, access policy
9.5.1 Making the invisible visible
This section explains how an Observability Harness reconstructs agent runs, correlates distributed evidence, and uses that evidence to improve governance decisions.
- Replayable agent trajectories: Explain why replaying a user prompt is insufficient and what must be preserved to examine an agent run later.
Why it matters: A recorded trajectory lets a team investigate what actually happened rather than assume the same prompt will recreate it. - Correlated governance evidence: Describe how the Observability Harness joins distributed signals into a causal record and distinguish execution records from evaluation and replay evidence.
Why it matters: Correlation makes it possible to connect an outcome to the agent, policy, approval, and actions that produced it. - Evidence-driven governance changes: Explain how collected evidence leads from a bad outcome to a corrective change and a new governance decision.
Why it matters: The harness has governance value when its evidence changes fixes and future controls, not merely when it stores records. - OpenTelemetry harness connection: Describe how the harness can use an existing telemetry pipeline while preserving the application-specific evidence it needs.
Why it matters: An existing telemetry backend can carry the signals, but it cannot supply the agent's governance context on its own.
Key terms: replay, agent trajectory, input references, Observability Harness, Orchestrate cell, causal record, correlation identifiers, logs, metrics, regression evaluation
9.5.2 The observability platform landscape
This section frames observability platform selection around durable evidence-path questions rather than changing feature lists or license labels.
- Observability platform selection criteria: Explain how to assess whether an observability platform can support a governed agent’s evidence path, and why a feature table, license label, or self-hosting claim is not enough.
Why it matters: These questions help you choose a platform that can preserve and connect the evidence your organization needs.
Key terms: telemetry, traces, evaluations, evidence path, self-hosting
9.5.3 What to observe: Metrics locate the problem; trajectories explain the path
The section shows how to use task, process, and latency metrics to locate problems, then use trajectories, source evidence, and calibrated quality judgments to investigate them.
- Metrics and trajectory evidence: Explain why investigation starts with task outcomes and cost, moves through process steps to model and API calls, and needs trajectories and source evidence to explain particular decisions.
Why it matters: This keeps a dashboard signal from being mistaken for an explanation of an agent’s decision. - Business-observable task success: Describe where to measure task success and why downstream use matters even though it is an incomplete signal.
Why it matters: An agent can produce a polished answer without accomplishing the user’s task. - Task cost and latency: Describe how to measure cost per task and latency so failed runs and long retry loops remain visible.
Why it matters: Failed retry loops can make an agent expensive and slow without standing out in an overall average. - First-attempt tool selection: Explain what tool selection accuracy measures and what a low rate can suggest.
Why it matters: A tool-selection problem can be investigated as a routing or capability-presentation problem rather than merely an unsuccessful task. - Human overrides and trust: Explain how human override rate informs decisions about trust levels and approval rules.
Why it matters: Override data can help adjust autonomy without ignoring either human disagreement or business outcomes. - Calibrated output quality judgment: Explain why the metrics need an explicit quality judgment and how to use an LLM-as-judge cautiously.
Why it matters: Operational metrics and automated scores can miss whether an output is genuinely good.
Key terms: metrics, trajectory, source evidence, access-controlled evidence store, task success rate, downstream use, cost per task, latency distribution, P50, P95
9.5.4 In production: Claude Code's observability stack
The section shows how Claude Code combines local session transcripts with exported observability signals, while leaving durable storage and action-level accountability to the deploying organization.
- Local transcript evidence: Explain what Claude Code session transcripts record and why a local transcript alone is not durable compliance evidence.
Why it matters: A replayable local record is useful, but it does not by itself satisfy long-term evidence needs. - Fleet signals and spans: Distinguish local transcripts from Claude Code's fleet-level metrics, structured events, and separate beta span-tracing path.
Why it matters: Fleet operations need signals that are easier to aggregate than individual conversation transcripts. - Organizational observability boundary: Explain what Claude Code supplies at the observability integration boundary and what a regulated deployment must add.
Why it matters: Exported telemetry becomes accountable operational evidence only when the organization stores it and connects actions to its own authorization context.
Key terms: session transcripts, JSONL, local replay artifact, retention, metrics, structured log events, span tracing, root span, redacted, integration boundary
9.5.5 Building it
This section builds a small teaching instrumentation layer that records agent activity, keeps sensitive evidence separate from telemetry, and distinguishes call results from task outcomes.
- Protected model call evidence: Explain what an LLM-call record captures and how an authorized investigator can inspect its messages and output without putting their contents in general telemetry.
Why it matters: Investigators can inspect the evidence for a call without exposing message bodies in ordinary telemetry. - Call results versus task outcomes: Distinguish model and tool-call results from the overall task outcome, including where approval overrides are preserved.
Why it matters: Separating these records prevents a working call from being mistaken for a successful task. - Teaching layer and production export: Describe the teaching instrumentation layer's span lifecycle and dashboard, and state what is needed to export its data for production observability.
Why it matters: The example shows what to instrument without implying that its dashboard and JSON export are a production observability system.
Key terms: messages_ref, output_ref, protected evidence store, general telemetry, tool span, task outcome, approval overrides, Span, AgentObserver, trace lifecycles
9.5.6 From evidence to improvement
Shows how to use an incident trace to diagnose an overcompensation failure, then turn the evidence into a tested, staged governance fix.
- Trace-based failure diagnosis: Trace an overcompensation incident from the coupon proposal through the evidence the model saw and the recorded policy decision. Distinguish a bad proposal from a gate that wrongly admitted it.
Why it matters: Locating the failure prevents a healthy-looking tool metric from masking an unauthorized outcome. - Evidence-driven governance repair: Describe how to convert the incident into regression tests and use their results to guide a staged fix. Include both the excessive proposal and a valid within-limit request.
Why it matters: Testing both denial and legitimate work makes a tighter boundary safer to deploy and easier to revise.
Key terms: trace, evidence references, policy decision, tool metric, regression case, trace ID, gate decision, business outcome, trust level
9.5.7 When it breaks
Explains two ways an observability harness can fail—too much trace data to review and traces that overstate what can be inferred—and how to address each.
- Trace volume and representative sampling: Explain why capturing every span may still leave an agent poorly observed, and describe how alerting and stratified sampling make important paths easier to find without favoring easy work.
Why it matters: Capturing more data does not help if important failures are buried or the traces you keep give a misleading picture. - Execution traces and reasoning limits: Distinguish a natural-language chain-of-thought from an execution record, and explain what observable traces can and cannot establish.
Why it matters: A trustworthy audit record must not mistake the model's explanation for proof of its internal reasoning.
Key terms: structured alerting, sampling, task difficulty, tenant, risk, complete retention, chain-of-thought, execution record, observable behavior, trace
9.6 Composing governance patterns
This section shows how four governance patterns coordinate around an action, how a host enforces task-wide limits, and how organizations scale governance in proportion to risk.
- Four-pattern governance pipeline: Describe how Approval Gate, Progressive Commitment, Blast Radius Control, and Observability Harness address different decisions about one proposed action.
Why it matters: The controls are complementary, not interchangeable checks. - Two-stage action authorization: Explain why unusable tools are removed during assembly while the selected tool and its arguments still require a runtime check against trusted task state.
Why it matters: A model's available choices and its account of the task cannot replace an enforceable authorization decision. - Task-wide cap enforcement: Apply the compensation example to explain why a proposal can fail a task-wide cap despite being individually approvable, and how a production gate protects that cap across concurrent proposals.
Why it matters: A per-call approval cannot prevent several otherwise acceptable actions from exceeding a shared limit. - Organizational control and accountability: Describe where governance controls live at organizational scale and why human approval alone does not establish accountability.
Why it matters: Enforcement and accountability need owners and shared infrastructure, not just an instruction or an approval button. - Risk-proportionate governance cost: Justify varying governance overhead according to an action's worst credible effect, using the section's examples.
Why it matters: Appropriate boundaries preserve useful autonomy while keeping consequential actions within organizational tolerance.
Key terms: Approval Gate, Progressive Commitment, Blast Radius Control, Observability Harness, causal record, assembly, runtime, normalized arguments, trusted task state, authoritative ledger
9.7 Governance standards and the regulatory landscape
Connects engineering governance controls to the organizational obligation to demonstrate compliance with evidence.
- Governance controls and evidence: Explain that regulation asks not just what governance controls an agent has, but which obligations those controls satisfy and what evidence the organization can produce when challenged.
Why it matters: Controls are easier to defend when the organization can show what they governed and what happened in practice.
Key terms: engineering primitives, agent registry, policy versions, approval records, execution traces, incident history
9.7.1 The emerging governance ecosystem
The section places agent governance in an emerging standards landscape and explains why compliance needs runtime policy, evidence, and identity-aware audit decisions.
- Governance Framework Roles: Distinguish the binding EU AI Act from broader risk-management frameworks and agent-specific guidance, including non-binding guidance for agentic finance.
Why it matters: Different sources call for different governance responses; they are not interchangeable compliance checklists. - Runtime Policy and Evidence: Explain why no single governance pattern proves compliance and describe the distinct evidence the four patterns contribute to a governed system.
Why it matters: A governance claim is more credible when the system can show how policy shaped decisions and what happened afterward. - Identity Beyond Agent Discovery: Explain why discovering an agent's identity is insufficient for governance and name the identities that must reach runtime policy and audit decisions.
Why it matters: Knowing which agent was discovered is not enough to attribute and govern a particular run.
Key terms: EU AI Act, NIST AI Risk Management Framework, ISO/IEC 42001, Model AI Governance Framework for Agentic AI, SAFR, Approval Gate, Blast Radius Control, Progressive Commitment, Observability Harness, policy
9.8 The Argus checkpoint: From code reviewer to governed system
This checkpoint shows how Argus applies governance to a tool call, where its current boundary falls short, and how governance lets an organization delegate trust safely.
- Argus governance boundary: Explain which Argus operations currently pass through the governance facade and why a production build must extend that boundary.
Why it matters: A governance boundary cannot control an operation that bypasses it. - Evidence-backed human review: Describe what the host preserves and presents when a governed tool call requires human approval.
Why it matters: A reviewer needs the proposed change and its supporting evidence, not an unexplained approval prompt. - Provisional ASK routing: Trace how the facade handles an ASK decision, an explicit DENY, and the tool call that proceeds.
Why it matters: This distinction prevents an approval waiver from becoming a waiver of policy or an explicit denial. - Governance as allocated trust: Explain how the four governance patterns allocate trust and how governance differs from perception at the model boundary.
Why it matters: Defined permissions, limited damage, and recorded outcomes make greater delegation possible without handing control to the model.
Key terms: governance facade, governance boundary, external effect, review_context, evidence_ref, normalized proposal, approval callback, policy reason, ASK, should_ask_human
9.9 Summary
9.10 References
This section lists sources cited in the chapter on governance.
Where this is used later
- Argus as a composition self-test (chapter 10) · Capability restriction and least privilege
- The trace localizes the handoff (chapter 11) · The role of trajectories in explaining an agent's path
Concepts explained in this chapter
- damage limitationA control that limits the scope or potential impact of an agent's harmful or erroneous actions.
- observability harnessA system that records and exposes execution traces for monitoring, diagnosis, and audit.
- governanceThe agent capability and control framework for enforcing policies, constraints, oversight, and accountability.
- approval gateA control point that determines whether an operation may proceed autonomously or requires human authorization.
- progressive commitmentAn agent pattern in which reversible evidence precedes consequential action and autonomy changes according to observed outcomes.
- autonomy-level classificationThe degree to which an agent can make decisions and take actions without human intervention or authorization.
- authorizationAuthorization is a decision about whether a proposed operation is permitted under applicable policy and delegated objectives.
- evidenceInformation that supports or substantiates a claim or decision.
- trustThe confidence an organization grants an agent to act within defined limits, informed by its observed outcomes.
- accountabilityThe ability to connect an outcome and the decisions that produced it to responsible agents and people.
- least agencyA principle of granting an agent only the autonomy needed to perform its delegated task.
- trajectoryA record of events and operations during a system run, arranged to show its progression and support diagnosis.
- progressive trust levelA level of confidence in an agent that determines how much routine approval work may be removed.
- sandboxingA sandbox is an isolated execution environment that restricts a system's access to resources and external effects.
- control planeA control plane is the part of a system that governs its operation by managing authoritative decisions, policies, or configuration.
- isolationIsolation is a technique for containing the effects of a failure or operation so they do not propagate across system boundaries.
- decision recordAn inspectable record of a decision or conclusion and its supporting evidence, assumptions, and rationale.
- LLM-as-a-judgeModel-based judging is the assessment of outputs using a model as the judge.
- human-in-the-loopAn approach in which a person intervenes in an automated process by supplying information, editing output, or judging a result.
- authorization decisionThe recorded outcome of applying a policy to an agent action.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
