liveBook · Manning

Chapter 9: Governance: Constraints that enable freedom

What you will learn in chapter 9 of Designing AI Agents

The introduction explains why agents with consequential capabilities need governance and how clear boundaries can make delegation possible.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why agents with consequential capabilities need governance and how clear boundaries can make delegation possible.

Key terms: governance, capability, blast radius, delegates choices, retains responsibility, clear boundary

9.1 What is governance? Autonomy without accountability is a liability

This section explains why agent governance must preserve useful autonomy while checking runtime proposals, their cumulative direction, and the consequences of mistakes.

Key terms: delegated objective, least agency, authorization, API key, role-based access control (RBAC), runtime, accountability, execution evidence, policy decision, containment

9.1.1 The governance spectrum

This section presents autonomy as a graded operating choice, bounded by firm limits and adjusted to the evidence of trust and the risk of the task.

Key terms: autonomy spectrum, execution authority, human oversight, operating arrangements, hard ceiling, working room, trust earned from observed outcomes, risk of the current task

9.1.2 Constraints enable freedom

The section explains how firm operating boundaries can reduce routine interruptions while keeping an agent’s actions subject to review.

Key terms: hard boundary, sandboxing, permission prompts, sandbox, approval policy, Automated Reasoning checks, formal policy, findings

9.2 Pattern: Approval Gate

Introduces the Approval Gate as a pre-execution decision that routes proposed actions to allow, deny, or human review, and explains why those boundaries need calibration.

Key terms: Approval Gate, human-in-the-loop (HITL), deny, allow, review, reversibility, impact, approval fatigue, allow boundary

9.2.1 Risk classification and the approval ladder

This section explains how an Approval Gate routes a payment proposal by risk while keeping the proposal stable enough for meaningful review before execution.

Key terms: approval fatigue, changed account, deny rule, allow rule, ask route, deny → allow → ask, frozen proposal, reviewing, revalidating

9.2.2 Classifying action risk

This subsection classifies action risk by impact and reversibility rather than by tool name, while giving explicit allow and deny rules precedence over the resulting defaults.

Key terms: impact, reversibility, action risk classification, explicit deny and allow rules

9.2.3 In production: Claude Code's permission modes

Compares production approval controls and shows why a permission mode, hook, sandbox, or pause point is only part of a governed approval process.

Key terms: permission modes, bypassPermissions, --dangerously-skip-permissions, external isolation, PreToolUse hooks, permissionDecision, normal permission flow, fail-closed gateway, sandbox layer, approval policy layer

9.2.4 Approval is a contract across time

This section explains how to keep a human approval tied to a specific intended action and its eventual execution, even when circumstances change during the wait.

Key terms: intent, approval, execution, durable intent, digest, resume coordinate, approval lifetime, business preconditions, idempotency_key, one-time execution token

9.2.5 Building it

This section builds a teaching Approval Gate that classifies proposed actions, applies ordered deny and allow policies, and records why each action proceeds or requires approval.

Key terms: risk_level, classify_risk, policy tier, CRITICAL, external effects, deny rules, allow rules, precedence, audit log, reason

9.2.6 When it breaks

This section explains how approval gates fail through reviewer fatigue and incomplete policy checks, and why approval cannot guarantee a safe result.

Key terms: approval fatigue, approval-to-rejection ratio, allow path, tool-name matching, arguments, resources, environment, regression suite, critical external effect

9.3 Pattern: Blast Radius Control

The section introduces Blast Radius Control as a way to limit the damage an action can cause even after it has been approved.

Key terms: Approval Gate, Blast Radius Control, task boundary, capabilities, execution boundary, quantitative limits, fault injection

9.3.1 Nested containment: Sandbox inside sandbox

This subsection explains why blast radius control needs nested boundaries beyond a sandbox.

Key terms: sandbox, blast radius control, capability restriction, execution isolation, damage limitation, Hierarchy topology

9.3.2 Layer 1: Capability restriction (least privilege)

This section explains how to give an agent only the capabilities its task requires and expand them only when evidence warrants it.

Key terms: capability restriction, least privilege, sandbox, scoped credentials, API-side authorization, narrowest capability, boundary, real blocker

9.3.3 Layer 2: Execution isolation (sandboxing)

This section explains how execution isolation limits an agent’s technical reach and how to choose an isolation boundary based on threats and operational trade-offs.

Key terms: threat model, shared-kernel containers, user-space kernel, microVM, full VM-based isolation, filesystem isolation, read access, network isolation, proxy, technical reach

9.3.4 Layer 3: Damage limitation

This section explains how damage-limitation controls contain authorized actions and why increasing an agent’s trust must never raise its hard blast-radius ceiling.

Key terms: damage limitation, rate limits, budget caps, time lock, cancellation window, revalidation, Saga-style rollback, compensating actions, blast radius, hard ceiling

9.3.5 Building it

This section shows how to assemble blast-radius checks around an execution function while leaving physical isolation to an external runtime.

Key terms: SandboxConfig, SandboxedExecutor, execute_fn, policy envelope, isolated runtime, Layer 1, Layer 3, estimated_cost, cumulative_cost, max_cost_per_task_usd

9.3.6 When it breaks

This section explains two ways Blast Radius Control can fail: supposedly separate defenses can share a point of compromise, and permitted actions can still cause damage.

Key terms: Blast Radius Control, defense in depth, sandbox, kill switch, microVM, legitimate-action damage, limiter, Progressive Commitment, dry run, validation

9.4 Pattern: Progressive Commitment

Progressive Commitment adds a time dimension to governance: an agent earns consequential actions through reversible checks within a run, while its autonomy changes across runs based on observed outcomes.

Key terms: Progressive Commitment, reversible checks, autonomy, harness, business outcome, promotion and demotion criteria, human overrides

9.4.1 Earning trust through evidence

Progressive Commitment uses checks before individual actions and accumulated operating evidence across sessions to grant bounded autonomy while retaining the ability to withdraw it.

Key terms: Progressive Commitment, staged verification, dry run, Chain topology, Blast Radius Control, trust ladder, staged trust escalation, production promotion, trust record, automatic demotion

9.4.2 Human sovereignty triggers

This section identifies when an agent must seek human input regardless of its earned trust level.

Key terms: human-sovereignty triggers, critical operations, insufficient evidence, ethical sensitivity, should_ask_human, trust level

9.4.3 Enforcing the current trust boundary

This subsection compares where guardrail mechanisms enforce a trust-level policy and distinguishes enforcement from deciding whether an agent has earned more trust.

Key terms: enforcement point, model traffic, formal policies, runtime policies, AgentCore Gateway, trust level

9.4.4 A configured baseline and an earned trust level

This section distinguishes a human-configured permission baseline from per-action hooks that check a proposed tool call before it runs.

Key terms: permission modes, configured baseline, dontAsk, bypassPermissions, PreToolUse hooks, frozen proposal, commitment chain

9.4.5 Building it

This section shows how recorded outcomes adjust an agent’s cross-session trust level, how that level affects approval routing, and what the example implementation lacks for production use.

Key terms: TrustManager.record_action, cumulative results, consecutive-error count, TrustMetrics, promotion, demotion, Observe, EscalationThresholds, Approval Gate, should_ask_human

9.4.6 When it breaks

This section explains how progressive commitment can fail when a probe no longer matches the action being committed or when promotion evidence does not represent the work ahead.

Key terms: within-run clock, probe, dry run, commit, snapshot, cross-run clock, stratified sampling, task difficulty, operation classes, promotion

9.5 Pattern: Observability Harness

Introduces the Observability Harness as a way to preserve connected evidence of an agent’s decisions and results so its actions can be debugged, evaluated, and audited.

Key terms: Observability Harness, accountability, inputs, permissions, actions, outcomes, trace boundary, correlation identifiers, retention period, access policy

9.5.1 Making the invisible visible

This section explains how an Observability Harness reconstructs agent runs, correlates distributed evidence, and uses that evidence to improve governance decisions.

Key terms: replay, agent trajectory, input references, Observability Harness, Orchestrate cell, causal record, correlation identifiers, logs, metrics, regression evaluation

9.5.2 The observability platform landscape

This section frames observability platform selection around durable evidence-path questions rather than changing feature lists or license labels.

Key terms: telemetry, traces, evaluations, evidence path, self-hosting

9.5.3 What to observe: Metrics locate the problem; trajectories explain the path

The section shows how to use task, process, and latency metrics to locate problems, then use trajectories, source evidence, and calibrated quality judgments to investigate them.

Key terms: metrics, trajectory, source evidence, access-controlled evidence store, task success rate, downstream use, cost per task, latency distribution, P50, P95

9.5.4 In production: Claude Code's observability stack

The section shows how Claude Code combines local session transcripts with exported observability signals, while leaving durable storage and action-level accountability to the deploying organization.

Key terms: session transcripts, JSONL, local replay artifact, retention, metrics, structured log events, span tracing, root span, redacted, integration boundary

9.5.5 Building it

This section builds a small teaching instrumentation layer that records agent activity, keeps sensitive evidence separate from telemetry, and distinguishes call results from task outcomes.

Key terms: messages_ref, output_ref, protected evidence store, general telemetry, tool span, task outcome, approval overrides, Span, AgentObserver, trace lifecycles

9.5.6 From evidence to improvement

Shows how to use an incident trace to diagnose an overcompensation failure, then turn the evidence into a tested, staged governance fix.

Key terms: trace, evidence references, policy decision, tool metric, regression case, trace ID, gate decision, business outcome, trust level

9.5.7 When it breaks

Explains two ways an observability harness can fail—too much trace data to review and traces that overstate what can be inferred—and how to address each.

Key terms: structured alerting, sampling, task difficulty, tenant, risk, complete retention, chain-of-thought, execution record, observable behavior, trace

9.6 Composing governance patterns

This section shows how four governance patterns coordinate around an action, how a host enforces task-wide limits, and how organizations scale governance in proportion to risk.

Key terms: Approval Gate, Progressive Commitment, Blast Radius Control, Observability Harness, causal record, assembly, runtime, normalized arguments, trusted task state, authoritative ledger

9.7 Governance standards and the regulatory landscape

Connects engineering governance controls to the organizational obligation to demonstrate compliance with evidence.

Key terms: engineering primitives, agent registry, policy versions, approval records, execution traces, incident history

9.7.1 The emerging governance ecosystem

The section places agent governance in an emerging standards landscape and explains why compliance needs runtime policy, evidence, and identity-aware audit decisions.

Key terms: EU AI Act, NIST AI Risk Management Framework, ISO/IEC 42001, Model AI Governance Framework for Agentic AI, SAFR, Approval Gate, Blast Radius Control, Progressive Commitment, Observability Harness, policy

9.8 The Argus checkpoint: From code reviewer to governed system

This checkpoint shows how Argus applies governance to a tool call, where its current boundary falls short, and how governance lets an organization delegate trust safely.

Key terms: governance facade, governance boundary, external effect, review_context, evidence_ref, normalized proposal, approval callback, policy reason, ASK, should_ask_human

9.9 Summary

9.10 References

This section lists sources cited in the chapter on governance.

Where this is used later

Concepts explained in this chapter

← Chapter 8: Collaboration: When one agent is not enough Chapter 10: Composing patterns: From map to terrain →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.