Chapter 8: Governing for Uncertainty
The introduction uses cloud computing’s governance history to frame how organizations can govern GenAI despite unresolved risks and changing standards.
After this chapter you will be able to
- Governance under uncertainty. Explain the governance response the introduction recommends instead of either rejecting GenAI until everything is solved or adopting it without controls.
- Whole-system governance decisions. Explain why approving a model is insufficient and describe the system-level decisions that make a use case possible to accept or reject honestly.
- Conditional ongoing approval. Explain why release approval applies to a particular system and environment at a particular time, and what continuing governance requires afterward.
- Prompt injection across connections. Explain why hostile text becomes a system-level risk for an autonomous agent and how controls limit, but cannot eliminate, its impact.
- Deletion beyond stored copies. Distinguish deletion of identifiable stored copies from removing personal data's influence on a trained model, and describe the architectural response.
- Accuracy and tolerable error. Explain why grounding and other accuracy controls help without ensuring a correct answer, and relate the remaining risk to the use case's tolerance for error.
- Workflow visibility versus model explanation. Distinguish records that make a system's workflow inspectable from claims of complete output explanation or training-data provenance.
- Agent objectives and bounded autonomy. Explain how an agent can pursue a stated objective through harmful unintended actions, why testing cannot reliably rule that out, and how architecture bounds the impact.
- Reusable governance standards. Explain how shared standards and infrastructure can turn governance requirements into testable controls and readable evidence, rather than leaving every team to build one-off solutions.
- Task scoped agent controls. Trace how a task-specific policy can constrain an agent before an action runs, and distinguish that check from asking a person to approve a sensitive action.
- Model independent governance infrastructure. Justify keeping long-lived governance controls in organization-owned systems rather than tying them to a particular model interface, while acknowledging the limits of emerging tools.
- Use Case Risk Questions. Describe how to review an AI system by examining what it can read and do, whose authority it uses, the potential damage before detection, and whether an incident can be reconstructed.
- Conditional Use Decisions. Explain why a review should produce specific operating conditions and allow approval, restricted approval, or a decision not to proceed yet.
- Connected Governance Practices. Trace how policy becomes operating controls, testing, monitoring, and renewed decisions when a system or its use changes.
- Present Day Organizational Accountability. Justify making decisions with today’s evidence despite unsettled AI controls, and explain who remains answerable when a system affects someone.
What it builds on
- The Six Level Governance Framework (chapter 3) · The governance framework for making decisions about a real system
- Tool Authorization (chapter 4) · Tool authorization for agents that can act
- Operations & Monitoring (Level 5) (chapter 3) · Operations and monitoring after launch
- Untrustworthy Input (chapter 4) · The problem of untrustworthy input.
- Making models forget and correct (chapter 5) · The difficulty of making models forget or correct information.
- Accuracy & Reliability (chapter 6) · Model accuracy and reliability failures.
- Transparency and Explainability (chapter 6) · Transparency and explainability concerns.
- Ethics in Agentic and Multi-Agent Deployments (chapter 6) · Ethical risks in agentic deployments.
Section by section
Introduction
The introduction uses cloud computing’s governance history to frame how organizations can govern GenAI despite unresolved risks and changing standards.
- Cloud governance analogy: Explain how organizations made cloud computing governable without making it risk free, and where the analogy with GenAI stops.
Why it matters: The cloud example shows why imperfect technology need not mean waiting for perfect governance. - Unresolved GenAI risks: Describe why current GenAI governance involves uncertainty rather than a complete set of technical fixes.
Why it matters: Recognizing unresolved risks keeps a governance plan from promising certainty its controls cannot provide. - Governance under uncertainty: Explain the governance response the introduction recommends instead of either rejecting GenAI until everything is solved or adopting it without controls.
Why it matters: This is the chapter’s central decision principle: act with explicit limits and accountability rather than pretending uncertainty can be avoided.
Key terms: cloud, GenAI, controls, evaluations, data filters, approval steps, logs, prompt injection, governance, uncertainty
8.1 What can you govern today
This section explains what organizations can govern today: the whole AI system, the consequences of changes to it, and the continuing validity of release decisions.
- Whole-system governance decisions: Explain why approving a model is insufficient and describe the system-level decisions that make a use case possible to accept or reject honestly.
Why it matters: A strong model or contract cannot compensate for unsafe access or a poorly scrutinized workflow. - Cross-cutting feature change review: Apply the section’s email-permission example to explain why a seemingly small agent change needs review across multiple concerns.
Why it matters: A minor-looking permission change can materially alter what information leaves the organization and who is accountable. - Conditional ongoing approval: Explain why release approval applies to a particular system and environment at a particular time, and what continuing governance requires afterward.
Why it matters: Monitoring and intervention keep a release decision meaningful when the live system or its effects change.
Key terms: system around it, authority, evidence, remaining risk, email third party recipients, privacy, security, auditability, accountability, Approval
8.2 What remains unsolved
This section distinguishes governance gaps organizations can fix from unresolved AI problems, then explains how to limit the impact of those problems without claiming to solve them.
- Organizational gaps versus open problems: Distinguish governance problems caused by an organization falling behind from problems that remain unsettled even when a careful team does the basics well.
Why it matters: The distinction prevents a team from treating an open technical problem as something it can fix by simply completing a checklist. - Prompt injection across connections: Explain why hostile text becomes a system-level risk for an autonomous agent and how controls limit, but cannot eliminate, its impact.
Why it matters: If an agent can be misled, its access and permissions determine how much damage it can do. - Deletion beyond stored copies: Distinguish deletion of identifiable stored copies from removing personal data's influence on a trained model, and describe the architectural response.
Why it matters: Designing for correction and deletion reduces dependence on an unlearning capability that cannot yet be reliably verified. - Accuracy and tolerable error: Explain why grounding and other accuracy controls help without ensuring a correct answer, and relate the remaining risk to the use case's tolerance for error.
Why it matters: A use case needs a margin for catching mistakes, not just measures that make answers look well supported. - Workflow visibility versus model explanation: Distinguish records that make a system's workflow inspectable from claims of complete output explanation or training-data provenance.
Why it matters: Precise claims let reviewers inspect real evidence without mistaking workflow records for complete insight into the model. - Agent objectives and bounded autonomy: Explain how an agent can pursue a stated objective through harmful unintended actions, why testing cannot reliably rule that out, and how architecture bounds the impact.
Why it matters: When an agent's behavior cannot be fully assured beforehand, bounded autonomy limits the consequences of an unintended path.
Key terms: governance problems, misalignment, machine unlearning, prompt injection, autonomous agent, tools and permissions, model weights, personal data, ground answers, error patterns
8.3 Better tools are arriving
The section shows how emerging standards and reusable tools can make GenAI governance more consistent and auditable, while explaining why organizations still need runtime controls, judgment, and model-independent infrastructure.
- Reusable governance standards: Explain how shared standards and infrastructure can turn governance requirements into testable controls and readable evidence, rather than leaving every team to build one-off solutions.
Why it matters: Reusable controls make governance easier to apply and review across systems. - Task scoped agent controls: Trace how a task-specific policy can constrain an agent before an action runs, and distinguish that check from asking a person to approve a sensitive action.
Why it matters: Checking task scope before execution keeps unrelated agent actions from becoming approval requests or reaching outside systems. - Documentation versus runtime controls: Distinguish what AI compliance platforms help teams document from what runtime controls must do when an agent acts.
Why it matters: A complete audit trail is not the same as preventing an unauthorized agent action. - AI assisted policy mapping: Describe how AI can help apply an existing privacy requirement across a large technical system while leaving the resulting evidence for human review.
Why it matters: Finding where a rule applies across a large system can be as difficult as stating the rule. - Model independent governance infrastructure: Justify keeping long-lived governance controls in organization-owned systems rather than tying them to a particular model interface, while acknowledging the limits of emerging tools.
Why it matters: Governance remains usable across model changes only if its durable controls do not depend on one model’s interface.
Key terms: testable control objectives, engineering evidence, standards, reusable tools, OpenTelemetry, MCP, task-specific policy, policy enforcer, human approval, Agent Governance Toolkit
8.4 The road ahead
The section shows how to make accountable, evidence-based decisions about AI use now, even while controls and rules continue to change.
- Use Case Risk Questions: Describe how to review an AI system by examining what it can read and do, whose authority it uses, the potential damage before detection, and whether an incident can be reconstructed.
Why it matters: A reassuring product label does not reveal the access or harm a system can create. - Conditional Use Decisions: Explain why a review should produce specific operating conditions and allow approval, restricted approval, or a decision not to proceed yet.
Why it matters: Specific limits make a decision usable in production and leave room to change it when evidence changes. - Connected Governance Practices: Trace how policy becomes operating controls, testing, monitoring, and renewed decisions when a system or its use changes.
Why it matters: A policy or launch review alone cannot keep governing a system as evidence and use change. - Sector Specific Evidence Duties: Explain how regulated-industry requirements fit into governance without being replaced by the framework.
Why it matters: A general governance process does not discharge a bank’s, hospital’s, insurer’s, or public agency’s specific obligations. - Present Day Organizational Accountability: Justify making decisions with today’s evidence despite unsettled AI controls, and explain who remains answerable when a system affects someone.
Why it matters: Unfinished technology does not postpone responsibility for decisions that affect real people.
Key terms: useful review, product category, untrusted content, operating conditions, least access, production evidence, delegation, 6L-G framework, technical controls, operating practices
8.5 Summary
The section summarizes how organizations can govern AI now while acknowledging unresolved problems and retaining responsibility for decisions about residual risk.
Concepts explained in this chapter
- AI governanceAI governance is the system of principles, processes, and oversight used to direct and control the development and use of artificial intelligence.
- accuracyAccuracy is the data-protection principle that personal data be correct and, where necessary, kept up to date.
- machine unlearningMachine unlearning is a set of methods for removing the influence of specified training data from a trained machine-learning model.
- misalignmentMisalignment is a mismatch between an AI agent’s measured or specified objective and the intended outcome.
- prompt injectionAn attack that uses instructions in content supplied to a model to redirect its behavior contrary to intended instructions.
- 6L-G modelThe book’s six-level framework for translating generative AI governance principles into organizational and technical controls.
- riskRisk is the possibility of an adverse outcome arising from uncertainty about an activity, system, or decision.
- contextual security research frameworkA contextual security research framework defines which actions are in scope for a task and checks proposed actions against task-specific rules.
- policy enforcerA policy enforcer is a software component that checks proposed actions against applicable rules before they proceed.
- generative artificial intelligenceGenerative AI is a class of artificial intelligence models that produces new content such as text, code, or images in response to inputs.
- principle of least privilegeLeast privilege is the security principle of granting a system or user only the minimum access needed to perform its task.
- provenanceThe documented origin and history of data, including how it was collected, transformed, and used.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
