liveBook · Manning

Governance

As explained in Designing AI Agents

The agent capability and control framework for enforcing policies, constraints, oversight, and accountability.

Governance defines the boundaries within which an agent may act when people or organizations delegate choices but retain responsibility for their effects. Clear boundaries can enable more autonomy: low-risk actions proceed, consequential proposals receive scrutiny, and mistakes are contained. section 9.0 section 9.9

Unlike a conventional program with largely predetermined paths, an agent may select tools, construct arguments, and combine operations after seeing the current context. Governance therefore surrounds the whole agent: it constrains what perception can access, what reasoning may consider, what actions may execute, and what reflection must flag. It also preserves evidence of decisions and outcomes. section 9.1 section 2.2.1

A proposal can be acceptable in isolation yet contribute to a sequence that drifts away from the delegated objective. Controls must consider both the current action and the trajectory of prior actions; approval of each small exception is not enough by itself. section 9.1

Four complementary controls follow an action from proposal to outcome:

  • Approval Gate evaluates the proposal and caller credentials, then routes the action to allow, deny, or human review. section 9.9
  • Blast Radius Control limits the maximum effect of an action, including one that was authorized but turns out to be wrong. section 9.9
  • Progressive Commitment expands or contracts autonomy as evidence accumulates, while retaining a hard ceiling at every trust level. section 9.9
  • Observability Harness connects decisions, execution, and outcomes so a run can be reconstructed and responsibility assigned. section 9.9

A governed tool call

Argus routes selected calls through a governance facade in the order gate, trust, policy envelope, and observation. A denial cannot be waived. A proposal routed to human review may proceed without asking only when the current scoped trust level permits it; critical external effects still require review. The checkpoint governs repository-read permission checks and lint actions, not every module call. A production system must route every operation with an external effect through the boundary. section 9.8

Listing 9.17 ArgusGovernance: The governed tool-call path (argus/governance.py)
    def run_tool(
        self, tool_name: str, args: dict,
        execute_fn, ask_human=None,
        cost: float = 0.01,
        review_context: dict | None = None,
    ) -> dict:
        """Gate -> trust -> policy envelope -> observe."""
        action = ToolAction(
            tool_name=tool_name, arguments=deepcopy(args)
        )
        decision, reason = self.gate.evaluate(action)
        self.observer.record_decision(           #A
            tool_name, decision.value, reason,
            evidence_ref=(review_context or {}).get(
                "evidence_ref"
            ),
            context={"scope": self.trust_scope,
                     "policy_version": (review_context or {}).get(
                         "policy_version")},
        )
        if decision is Decision.DENY:
            return {"error": reason}
 
        if decision is Decision.ASK and (
            self.trust.should_ask_human(
                action.risk_level.value
            )
        ):
            required = {  #B
                "objective", "target", "change",
                "evidence_ref", "policy_version",
            }
            if any(not (review_context or {}).get(k)
                   for k in required):
                self.observer.record_decision(
                    tool_name, "review_context_missing",
                    "Required approval evidence is absent",
                )
                return {"error": "Review context incomplete"}
            packet = {
                "tool": action.tool_name,
                "arguments": deepcopy(action.arguments),
                "risk": action.risk_level.value,
                "reversible": action.reversible,
                "policy_reason": reason,
                "context": deepcopy(review_context or {}),
            }
            reviewer_id = (
                ask_human(packet) if ask_human else None
            )
            if not isinstance(reviewer_id, str) or (
                not reviewer_id.strip()
            ):
                self.trust.record_action(
                    success=False, user_override=True
                )
                self.observer.record_decision(
                    tool_name, "declined", reason
                )
                return {"error": "Declined"}
            self.observer.record_decision(
                tool_name, "human_approved",
                f"{reason}; reviewer={reviewer_id}; "
                f"scope={self.trust_scope}",
            )
        elif decision is Decision.ASK:  #C
            status = self.trust.get_status()
            self.observer.record_decision(
                tool_name, "approval_waived",
                f"{reason}; trust={status['level']}; "
                f"scope={self.trust_scope}",
            )
        start = time.time()
        result = self.sandbox.execute(           #D
            tool_name, action.arguments, cost, execute_fn
        )
        ok = "error" not in result
        self.observer.record_tool_call(
            tool_name, ok,
            (time.time() - start) * 1000,
        )
        self.trust.record_action(success=ok)     #E
        return result
 
    def finish_review(
        self, success: bool,
        human_override: bool = False,
    ) -> dict:
        self.observer.record_task_outcome(
            success, human_override
        )
        self.observer.finish_trace(success=success)
        return self.observer.get_dashboard()

Inspect the governed tool-call path, including the recorded gate decision and the ordering of controls. section 9.8

Human approval needs a reviewable evidence packet: objective, target, current and proposed change, evidence reference, and policy version. The supporting packet is retained in a protected, versioned store, and an approval records the reviewer’s identity. Governance evidence can also include execution traces, incident history, and the basis for autonomy changes. No single control proves compliance; applicable obligations depend on the use case and organizational role. section 9.8 section 9.9 section 9.7.1

How Designing AI Agents covers it

How deep it goes
Develops four complementary controls and a governed tool-call path, while identifying operations outside the checkpoint boundary.
Examples
Uses a screen-time negotiation to illustrate drift and an Argus tool-call facade to demonstrate enforcement and review.
What you can do after
Apply governance by routing proposals through approval, trust, policy limits, and observation, while identifying gaps in boundary coverage.
How it connects
Connects governance to perception, reasoning, action, reflection, collaboration, and memory as distinct architectural responsibilities.

Also explained in

Compare how 9 books explain governance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.