Chapter 1: Governing Generative
The chapter introduction explains why GenAI’s rapid, flexible adoption makes practical governance urgent.
After this chapter you will be able to
- GenAI’s Expanded Risk Exposure. Explain why GenAI strains older governance approaches: people can use it instantly across many domains without developer involvement, and its benefits come with hard-to-predict failures.
- Practical Governance Over Checklists. Explain why the introduction calls for effective, practical GRC rather than treating governance as a compliance box, including why early governance work may save effort later.
- GenAI Governance Urgency. Explain why conventional IT controls and a fix-it-later approach are insufficient for GenAI, and what a GenAI-specific GRC program can realistically do.
- Model Attack Surfaces. Distinguish the prompt-based and training-pipeline attacks described here, and explain why an AI system that takes actions raises the stakes of prompt injection.
- Personal Data Lifecycle Risks. Explain the distinct privacy problems of repurposing personal data, reproducing memorized training data, and trying to erase data already learned by a model.
- Governance direction and accountability. Describe how AI governance connects external expectations, leadership decisions, and operational controls, including clear responsibilities for model builders, integrators, and users.
- Continuous AI risk management. Explain why AI risks must be repeatedly identified, assessed, and addressed across contexts, with shared responsibilities and decisions about which risks are acceptable.
- Compliance and trustworthiness gap. Distinguish meeting applicable laws, internal standards, and public commitments from demonstrating that an AI system actually behaves safely and as intended.
- Lifecycle embedded GRC. Explain why governance, risk, and compliance considerations should begin with an AI idea and remain part of design, deployment, and updates rather than being checked only at the end.
- Governance Under GenAI Uncertainty. Explain why traditional GRC assumptions are insufficient for GenAI and why governance must shift toward probabilistic risk management, continuous monitoring, and adaptable policies.
- Layered Controls and Ownership. Describe how GenAI GRC builds on classic GRC and AI GRC, and distinguish what model developers, integrators, and end-user organizations are responsible for.
- Risk Informed Project Decisions. Describe why the six-level model is both risk-informed and lifecycle-based, and explain what Strategy & Policy and Risk & Impact Assessment decide before a project proceeds.
- Design and Launch Gates. Trace how assessed risks become documented design controls and how Acceptance Testing checks those mitigations before release to real users.
- Operational Oversight and Improvement. Describe how governance continues after launch through monitoring, assigned operational ownership, end-of-life checks, and feedback into revised policies or engineering work.
- Anticipation and resilience in governance. Explain why the examples call for ongoing GenAI governance rather than a one-time compliance checklist.
- Policy and impact foundations. Explain how Strategy & Policy establishes accountability and how Risk & Impact Assessment identifies a project's legal obligations and risks to people and society.
- Threat modeling and design. Describe how Implementation Review uses AI threat modeling to identify possible failures or attacks and inform system design and documentation.
- Mitigation validation before launch. Explain how Acceptance Testing checks whether promised mitigations withstand adversarial and real-world conditions.
- Continuous operational oversight. Describe how live-system screening, drift detection, and records support ongoing oversight and investigation.
- Measured governance feedback loop. Explain how Learning & Improvement uses measured signals and other feedback to investigate changes and revise governance practices.
- Urgency and Opportunity of Governance. Explain why organizations should establish GenAI governance now and how doing it well can benefit them beyond compliance.
Section by section
Introduction
The chapter introduction explains why GenAI’s rapid, flexible adoption makes practical governance urgent.
- GenAI’s Expanded Risk Exposure: Explain why GenAI strains older governance approaches: people can use it instantly across many domains without developer involvement, and its benefits come with hard-to-predict failures.
Why it matters: The scale and flexibility that make GenAI useful also make its risks harder to contain with older approaches. - Practical Governance Over Checklists: Explain why the introduction calls for effective, practical GRC rather than treating governance as a compliance box, including why early governance work may save effort later.
Why it matters: Governance is useful when it changes everyday practice before failures become costly crises.
Key terms: GenAI, old governance playbook, Governance, Risk, and Compliance (GRC), day-to-day practices, compliance box
1.1 Why Governance, Risk and Compliance (GRC) for GenAI Matters Now?
This section explains why GenAI’s fast, hard-to-predict failures and distinctive security, privacy, and ethical risks make early, ongoing governance necessary.
- GenAI Governance Urgency: Explain why conventional IT controls and a fix-it-later approach are insufficient for GenAI, and what a GenAI-specific GRC program can realistically do.
Why it matters: It explains why governance has to begin before a GenAI failure becomes widespread or costly. - Upstream Governance Accountability: Describe how upstream providers’ choices can create downstream harm and how regulation and buyers can hold them accountable.
Why it matters: A vendor cannot assume that governance is solely its customer’s responsibility. - Model Attack Surfaces: Distinguish the prompt-based and training-pipeline attacks described here, and explain why an AI system that takes actions raises the stakes of prompt injection.
Why it matters: The entry point and potential harm change depending on whether an attacker controls a prompt, content the model encounters, or training examples. - Output Based Model Extraction: Describe how model extraction can copy a model’s behavior without access to its code or weights, and why it may be difficult to notice.
Why it matters: An organization can lose the value of a model while its service appears to be operating normally. - Personal Data Lifecycle Risks: Explain the distinct privacy problems of repurposing personal data, reproducing memorized training data, and trying to erase data already learned by a model.
Why it matters: Privacy decisions made when data is collected and used can have consequences that are difficult to reverse after training. - Bias and Public Trust: Explain how biased data and model outputs can lead to discriminatory deployments and weaken trust in AI.
Why it matters: Bias is not confined to offensive outputs; deployed systems can turn it into consequential decisions.
Key terms: GenAI-specific GRC programme, hallucinations, move fast and fix later, upstream actors, downstream harms, EU AI Act, general-purpose AI, attack surface, jailbreaks, prompt injections
1.2 GRC for AI: More Than a Compliance Checklist
This section explains how governance, risk management, and compliance work together as proportionate, continuous practices built into AI development and use rather than treated as a final checklist.
- Proportionate AI oversight: Explain how an organization can make GenAI GRC practical by embedding oversight in existing routines, starting with its most significant risks, and scaling the effort as AI use grows.
Why it matters: This makes oversight feasible even when an organization cannot support a large formal governance program. - Governance direction and accountability: Describe how AI governance connects external expectations, leadership decisions, and operational controls, including clear responsibilities for model builders, integrators, and users.
Why it matters: Clear direction and ownership help prevent teams from making conflicting decisions about acceptable AI risks. - Continuous AI risk management: Explain why AI risks must be repeatedly identified, assessed, and addressed across contexts, with shared responsibilities and decisions about which risks are acceptable.
Why it matters: A system that performs well in a lab can still encounter unexpected failures in real-world use. - Compliance and trustworthiness gap: Distinguish meeting applicable laws, internal standards, and public commitments from demonstrating that an AI system actually behaves safely and as intended.
Why it matters: An organization can satisfy a documentation requirement while leaving a consequential safety or data-handling problem unresolved. - Lifecycle embedded GRC: Explain why governance, risk, and compliance considerations should begin with an AI idea and remain part of design, deployment, and updates rather than being checked only at the end.
Why it matters: Early, adaptable oversight matters when a harmful AI decision cannot simply be reversed after release.
Key terms: proportionality, existing risk management, IT governance, product development routines, governance, leadership roles, operational controls, shared responsibilities, accountability, risk management
1.3 A Mental Model for GenAI GRC
This section explains why GenAI needs adaptive governance and introduces a six-level model that carries risk decisions and oversight through the system lifecycle.
- Governance Under GenAI Uncertainty: Explain why traditional GRC assumptions are insufficient for GenAI and why governance must shift toward probabilistic risk management, continuous monitoring, and adaptable policies.
Why it matters: The governance approach has to keep working as the system’s behavior and risks change. - Layered Controls and Ownership: Describe how GenAI GRC builds on classic GRC and AI GRC, and distinguish what model developers, integrators, and end-user organizations are responsible for.
Why it matters: A layered view helps teams preserve useful existing controls without overlooking GenAI-specific risks or unclear ownership. - Risk Informed Project Decisions: Describe why the six-level model is both risk-informed and lifecycle-based, and explain what Strategy & Policy and Risk & Impact Assessment decide before a project proceeds.
Why it matters: The first decisions can prevent an unsuitable or insufficiently governed AI project from advancing. - Design and Launch Gates: Trace how assessed risks become documented design controls and how Acceptance Testing checks those mitigations before release to real users.
Why it matters: A risk decision matters only if its controls are built and shown to work before release. - Operational Oversight and Improvement: Describe how governance continues after launch through monitoring, assigned operational ownership, end-of-life checks, and feedback into revised policies or engineering work.
Why it matters: Controls need owners and revision after launch, including when a system is retired. - Erasure Across Governance Levels: Trace how a right-to-erasure request can create work across multiple governance roles and explain why the six-level mapping makes that work repeatable.
Why it matters: The erasure example shows how a broad obligation becomes assignable work rather than just a policy statement.
Key terms: GRC, probabilistic risk management, continuous monitoring, living governance frameworks, classic GRC, AI GRC, GenAI GRC, residual risk, supply chain, six-level governance (6L-G) model
1.4 Challenges in Practice: Illustrative Scenarios
Three GenAI incidents show how gaps in policy, assessment, testing, and monitoring can turn foreseeable risks into customer, security, and safety failures.
- Anticipation and resilience in governance: Explain why the examples call for ongoing GenAI governance rather than a one-time compliance checklist.
Why it matters: The cases show why safeguards must remain effective after a system is deployed. - Bank copilot advice failure: Trace how the bank copilot’s invented fee rule reached a customer and identify the governance gaps the example associates with it.
Why it matters: An AI suggestion can become a customer-facing promise when staff lack checks on its reliability. - API model extraction warning: Describe how partner API access exposed the startup’s model and why existing logs did not prevent the loss.
Why it matters: Recording suspicious API use offers little protection if no one detects and responds to it. - My AI child safety gaps: Explain how the My AI example links unsafe responses to failures across several governance layers.
Why it matters: Calling a widely deployed system experimental does not replace safeguards for vulnerable users.
Key terms: GenAI risks, anticipation and resilience, 6L-G model, GenAI copilot, hallucinated, Strategy & Policy, Acceptance Testing, Operations & Monitoring, model extraction attack, API logs
1.5 Tools and Practices You’ll Need
The section maps practical governance tools and practices to six levels of GenAI governance, from setting policy and assessing risks through testing, monitoring, and improvement.
- Policy and impact foundations: Explain how Strategy & Policy establishes accountability and how Risk & Impact Assessment identifies a project's legal obligations and risks to people and society.
Why it matters: These foundations make risks and responsibilities explicit before technical work proceeds. - Threat modeling and design: Describe how Implementation Review uses AI threat modeling to identify possible failures or attacks and inform system design and documentation.
Why it matters: Anticipated threats can shape safeguards before a system is released. - Mitigation validation before launch: Explain how Acceptance Testing checks whether promised mitigations withstand adversarial and real-world conditions.
Why it matters: A planned safeguard is not reliable until it has been tested under relevant conditions. - Continuous operational oversight: Describe how live-system screening, drift detection, and records support ongoing oversight and investigation.
Why it matters: Operational evidence helps teams notice problems and reconstruct what happened. - Measured governance feedback loop: Explain how Learning & Improvement uses measured signals and other feedback to investigate changes and revise governance practices.
Why it matters: Measured feedback helps governance adapt rather than rely on impressions or remain static.
Key terms: Strategy & Policy, Risk & Impact Assessment, impact assessment, Implementation Review, AI Threat modeling, model cards, system cards, Acceptance Testing, red teams, bias testing
1.6 Motivation and What’s Ahead
This section explains why organizations should act on GenAI governance now, both to reduce risks and to support responsible innovation.
- Urgency and Opportunity of Governance: Explain why organizations should establish GenAI governance now and how doing it well can benefit them beyond compliance.
Why it matters: Governance can help an organization manage immediate risks while making responsible AI a lasting strength.
Key terms: GenAI GRC, governance frameworks, sustainable innovation, public trust
1.7 Summary
The section summarizes GenAI governance as an ongoing, cross-functional discipline supported by a lifecycle-spanning framework.
Where this is used later
- The Six Level Governance Framework (chapter 3) · Introduces a governance framework for managing risks across the AI lifecycle.
- Risk dimensions across deployment postures (chapter 2) · Compares risk across deployment postures with different degrees of organizational and vendor control.
- Provider, deployer, and other roles (chapter 7) · Examines provider and deployer roles in more detail.
- Untrustworthy Input (chapter 4) · Examines untrustworthy input in more depth.
- The Ability to Make Changes (chapter 4) · Examines risks when AI can make changes.
- Backdoors, Poisoning and Malicious Models (chapter 4) · Examines poisoning and malicious models.
- Model Theft (chapter 4) · Covers model theft as a security risk.
- Collection and Purpose (chapter 5) · Develops collection, lawful purpose, and reuse risks.
Concepts explained in this chapter
- GRC programAn organizational program that integrates governance, risk management, and compliance activities for a technology or business area.
- hallucinationAn AI hallucination is an output that presents fabricated, unsupported, or false information as though it were valid.
- prompt injectionAn attack that uses instructions in content supplied to a model to redirect its behavior contrary to intended instructions.
- 6L-G modelThe book’s six-level framework for translating generative AI governance principles into organizational and technical controls.
- monitoringObservation of a system or process to track its status, behavior, or performance.
- probabilistic risk managementProbabilistic risk management evaluates and governs risks whose likelihoods or outcomes are uncertain rather than deterministic.
- risk and complianceGovernance, risk, and compliance is the coordinated management of organizational oversight, risk controls, and compliance obligations.
- AI governanceAI governance is the system of principles, processes, and oversight used to direct and control the development and use of artificial intelligence.
- ai risk managementAI risk management is the continuous process of identifying, assessing, addressing, and monitoring risks posed by AI systems to organizations and stakeholders.
- complianceCompliance is conformity with applicable laws, regulations, standards, policies, and contractual requirements.
- threat modelA structured technique for identifying, analyzing, and prioritizing threats to a system.
- generative artificial intelligenceGenerative AI is a class of artificial intelligence models that produces new content such as text, code, or images in response to inputs.
- personal dataInformation that can identify a specific person, either by itself or when combined with other information.
- anonymizationAnonymization is the alteration of information to prevent it from being linked to an identifiable person.
- attack surfaceThe set of points through which an attacker can attempt to affect or compromise a system.
- memorizationThe retention of specific training-data details by a model, potentially allowing it to reproduce them.
- prompt-based attackAn attack that manipulates an AI system through crafted natural-language instructions or content.
- training datasetThe collection of examples used to optimize a machine-learning model during training.
- learning and improvementThe Learning & Improvement level of the 6L-G model uses feedback and flaws found in live systems to improve AI governance, designs, processes, tests, and controls.
- strategy and policy levelThe 6L-G governance level that establishes organizational direction, AI policies, leadership, accountability, and oversight.
- data minimization strategyData minimization is the practice of limiting personal or sensitive data collection and use to what is necessary for a defined purpose.
- impact assessmentAn impact assessment is a documented evaluation of the risks an AI system introduces for individuals and society, recorded here as a structured review of a GenAI proposal.
- privacy-by-designPrivacy by design is an approach that builds privacy protections into the design and operation of systems from the outset.
- robustnessRobustness measures how consistently a model behaves appropriately when its inputs vary or differ from familiar contexts.
- lifecycle-based governanceA living governance structure that embeds controls and oversight across the stages of an AI system's existence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
