Chapter 3: The Six Level GenAI Governance Framework
This introduction uses three incidents to show why GenAI governance needs multiple layers beyond written policies, approved vendors, or a single security tool.
After this chapter you will be able to
- Layers of GenAI governance. Explain the three governance layers illustrated by the incidents: controlling which tools handle sensitive data, configuring and monitoring approved tools, and ensuring safety problems reach people who can respond.
- Six Level Governance Lifecycle. Describe what each of the six levels does, from setting policy and assessing a use case through design review, launch approval, monitoring, and improvement based on findings.
- Three Governance Maturity Buckets. Distinguish Insufficient, Baseline, and Advanced governance by their processes, accountability, controls, and response to incidents.
- Policy boundaries and ownership. Explain what Strategy and Policy establishes before an AI feature is built, using MediAssist’s patient-facing capabilities as an example. Distinguish these decisions from the clinical rules and implementation details decided later.
- Charter and decision ownership. Explain how the Responsible AI Charter gives organizational principles an executive mandate, and distinguish the review function’s routine authority from the Steering Committee’s escalation role.
- Practical policies and existing processes. Distinguish the Acceptable Use Policy from the review policy, and explain why AI governance should adapt existing privacy, security, and business-risk processes.
- Impact-based risk tiering. Explain how risk tiers make reviews consistent, why impact takes priority over whether a tool is internal or public, and why agents need a separate check within tiering.
- AI inventory and discovery. Describe what the AI inventory connects for each running system, what additional agent details it records, and why it must surface unauthorized use.
- Charter rule and authority. Explain the charter’s rule for clinical AI and the RAI Office’s authority over proposed uses.
- Patient-facing product limits and approvals. Describe what MediAssist’s acceptable use policy permitted and prohibited, and the two decisions required before work began on features that directly discussed health concerns with patients.
- Feature-Specific Risk Discovery. Explain why a feature that appears to fit policy still needs an AI Impact Assessment, using Derm-Cam to illustrate risks revealed by concrete questions.
- Conditional Assessment Decision. Describe the decision an AI Impact Assessment should produce and how conditions and evidence shape whether Derm-Cam can move forward.
- AIIA scope and questions. Explain what an AIIA records and why its questions must be answered early enough to shape the design. Describe the kinds of questions it asks about affected people, data, model evaluation, risks, controls, post-launch responsibilities, and external tools.
- Actionable assessment findings. Describe how AIIA findings become concrete requirements or obligations, and how each finding is tracked.
- Risk-triggered follow-up reviews. Describe how elevated risks found by the AIIA can trigger additional reviews and how their results stay connected to the proposal.
- Assessment Process Effectiveness. Explain why routing every proposal through the AIIA template is not enough, and describe what the RAI Office and Steering Committee need to track to see whether the review process works.
- Derm-Cam assessment findings. Describe the three concerns identified for Derm-Cam: identifiable skin photos that may be health data, vendor retention exceeding policy, and untested performance across skin tones.
- Assigned risk mitigation work. Explain how the assessment assigned responses to the findings, including legal review, changed retention terms, skin-tone testing, clinician fallback, and approved patient-facing messages.
- Traceable conditional approval. Explain how MediAssist made findings verifiable and why Derm-Cam received only conditional approval.
- Early architecture risk review. Explain why design and implementation reviews start before the system is built, using Derm-Cam’s direct vendor API call as an example of a convenient choice that may introduce risk.
- Practical control requirements. Explain how a Level 3 review should handle requirements that are sound in principle but infeasible or disproportionately costly in the actual system.
- Review findings and release tracking. Describe how review decisions and Level 2 requirements are recorded and carried into engineering work, and distinguish those release records from the AI inventory.
- AIIA conditions in implementation artifacts. Trace an AIIA condition to the design, backlog work, and test that will implement and prove it before launch, and describe how reviewers check the requirements they own.
- Consistent yet flexible technical standards. Explain how technical standards make implementation reviews consistent and why those standards must accommodate technical infeasibility.
- Implementation Review Metric Purpose. Explain that Level 3 metrics should show whether technical reviews happen early enough and whether their findings become work, using records organizations already have.
- Architecture Trace to Build Controls. Explain how tracing Derm-Cam’s flow showed where its AIIA conditions had to be implemented, including retention, clinician review, and patient-facing wording.
- Photo Identifier Privacy Controls. Describe how uploaded photos could collect unintended personal data and how Derm-Cam changed the upload, processing, and logging flow to limit it.
- Image Text Injection Boundary. Trace how OCR could turn text hidden in an uploaded image into an instruction to the LLM, and explain Derm-Cam’s resulting response constraints.
- Implemented Control Verification. Explain why Launch Readiness Review verifies that required controls are working, rather than accepting implementation tickets as proof. Describe the kinds of checks illustrated by Derm-Cam, including image metadata removal, approved patient wording, required reviews and tests, retention settings, and whether the tested build and configuration match the planned release.
- Risk Scaled Review Approval. Distinguish how the review and approval of LRR can differ between low-risk and high-risk systems.
- Failed Check Launch Gate. State the permitted responses when a required LRR check fails and explain when the feature must not launch.
- Release Candidate Evidence Quality. Explain why a generic completed test or vendor claim may be inadequate LRR evidence, and describe the evidence Derm-Cam needs for adversarial and skin-tone performance checks.
- Launch review evidence set. State what the product owner and required approvers need to know for a launch readiness review: what was tested, which version, what failed, what was fixed, and who approved the launch.
- Launch readiness metric checks. Explain what the launch readiness metrics check: timely required reviews, appropriate evidence, and the handling of failed tests.
- Release Candidate Approval Gate. Explain how Derm-Cam’s launch decision was tied to the exact release candidate and the conditions set by earlier reviews. Describe what the release record contained and when the steering committee approved rollout.
- Risk Appropriate Launch Reviewers. Distinguish the lighter approval path described for a low-risk chatbot from Derm-Cam’s named-reviewer approach, and describe what each reviewer checked.
- Need for post-launch governance. Explain why launch approval alone cannot control a live AI application: its models, settings, connections, and user requests can change after testing.
- Configuration and request records. Distinguish the system configuration record from the request record and explain how each supports investigation without retaining unnecessary personal data.
- Baseline-based change review. Explain how significant model, prompt, data-source, service-connection, or safety-setting changes should be evaluated before users receive them.
- Findable failure evidence. Explain why operations and monitoring records need an assigned owner and location.
- Uncaught pilot safety breach. Describe how Derm-Cam violated its approved rules during the pilot and how the problem was discovered.
- Missing investigation evidence. Explain why the available record could not establish what had changed or which safeguard failed.
- Pilot containment and incident record. Describe the team's immediate response and what it specified for future investigations.
- Findings into lasting controls. Explain why a finding should change the failed checklist, template, test, monitoring rule, or product limit rather than remain only in an incident ticket.
- Limits and closure decisions. Explain when learning may require limiting, pausing, or retiring a product, and what the decision record should capture.
- Findings That Change Artifacts. Explain why logging incidents and holding retrospectives are not enough: a finding produces improvement only when it changes the artifacts that govern system behavior.
- Level 6 follow-through metrics. Explain how Level 6 measures follow-through on a finding: whether the owner closed the fix on time, whether the issue returned, and whether the finding changed the control that missed it.
- Contextual Action Policies. Explain why an agent's actions need conditions based on its live state rather than blanket tool permissions, and describe how rules can be checked before each action.
- Delegation Authority Trails. Explain why sequences of actions and multi-hop delegation require both a complete requester trail and narrowing authority at each handoff.
- Governance of Agent Memory. Describe the privacy obligations created when agent memory persists across runs and how MediAssist's memory design supports isolation and deletion.
- Runtime Control Capabilities. Distinguish pre-action policy enforcement, identity brokering, and logging as capabilities needed to govern agents, regardless of vendor.
- Agent Inventory and Visibility. Describe how to establish an agent registry and find active agents that did not enter through formal registration, then observe whether they use the control plane.
What it builds on
- MediAssist: A case study in AI governance (chapter 2) · Introduces MediAssist as the case study.
- What Does “Agentic” Mean? (chapter 2) · The distinction between an agent that may act and a system that only answers.
- SaaS or Application Consumers (chapter 2) · The SaaS consumer posture provides context for a vendor that may not support gateway routing.
- Agentic AI (chapter 2) · The concept of agents acting with autonomy.
Section by section
Introduction
This introduction uses three incidents to show why GenAI governance needs multiple layers beyond written policies, approved vendors, or a single security tool.
- Layers of GenAI governance: Explain the three governance layers illustrated by the incidents: controlling which tools handle sensitive data, configuring and monitoring approved tools, and ensuring safety problems reach people who can respond.
Why it matters: An approved AI tool can still expose sensitive data, behave badly, or cause safety problems if the surrounding controls and response processes are missing.
Key terms: GenAI governance, guardrails, live monitoring, data loss prevention (DLP), safety problems
3.1 The Six Level Governance Framework
This section introduces the six governance levels, defines recurring terms, and explains how governance maturity varies by organization and use case.
- Six Level Governance Lifecycle: Describe what each of the six levels does, from setting policy and assessing a use case through design review, launch approval, monitoring, and improvement based on findings.
Why it matters: The lifecycle shows where decisions are made before launch and how experience after launch changes them. - Recurring Governance Terms: Distinguish the section’s definitions of a threat model, prompt injection, jailbreak, hallucination, drift, and regression test.
Why it matters: These definitions make later discussions of risks, reviews, and monitoring easier to follow. - Three Governance Maturity Buckets: Distinguish Insufficient, Baseline, and Advanced governance by their processes, accountability, controls, and response to incidents.
Why it matters: The buckets provide a way to describe whether governance is merely reactive, established, or proactive. - Context Dependent Governance Maturity: Explain why an organization need not reach the same maturity at every level or for every GenAI use case.
Why it matters: Maturity should be judged against what the organization is building and using, not as a uniform score across all six levels.
Key terms: Six Level GenAI Governance (6L-G), Level 1, Level 2, Level 3, Level 4, Level 5, threat model, prompt injection, jailbreak, hallucination
3.2 Strategy & Policy (Level 1)
This section explains how Strategy and Policy sets AI-use boundaries and approval responsibilities before teams design specific safeguards.
- Policy boundaries and ownership: Explain what Strategy and Policy establishes before an AI feature is built, using MediAssist’s patient-facing capabilities as an example. Distinguish these decisions from the clinical rules and implementation details decided later.
Why it matters: Teams need clear boundaries and ownership before they build capabilities that affect patients. - Consequences of weak policy: Describe how weak Strategy and Policy leaves teams making inconsistent AI-use decisions, even when governance tools or vendor safeguards exist.
Why it matters: A concrete policy makes decisions about AI use consistent across teams.
Key terms: Strategy and Policy, formal approval, exception, human must stay in the loop, named owner, unauthorized AI chatbot, governance tools, vendor safeguards, concrete policy
3.2.1 Key artifacts
This section explains the artifacts and checks that turn an organization’s GenAI principles into approval decisions, tracked deployments, and observable governance.
- Charter and decision ownership: Explain how the Responsible AI Charter gives organizational principles an executive mandate, and distinguish the review function’s routine authority from the Steering Committee’s escalation role.
Why it matters: Clear authority keeps approvals from stalling or being made by people who do not own the risk. - Practical policies and existing processes: Distinguish the Acceptable Use Policy from the review policy, and explain why AI governance should adapt existing privacy, security, and business-risk processes.
Why it matters: Teams need both clear boundaries and a workable route to approval without a parallel bureaucracy. - Impact-based risk tiering: Explain how risk tiers make reviews consistent, why impact takes priority over whether a tool is internal or public, and why agents need a separate check within tiering.
Why it matters: Tiering makes approval requirements depend on a use’s consequences rather than a reviewer’s preferences or a tool’s visibility. - AI inventory and discovery: Describe what the AI inventory connects for each running system, what additional agent details it records, and why it must surface unauthorized use.
Why it matters: Governance cannot track risks in systems or agents it has not identified. - Governance effectiveness metrics: Explain why a signed charter is insufficient evidence of adoption and describe the kinds of metrics governance leaders should track.
Why it matters: These signals reveal gaps between a governance policy and what teams actually do.
Key terms: Responsible AI Charter, Responsible AI Office, GenAI Steering Committee, Acceptable Use Policy, review policy, intake questionnaire, risk tiers, impact, agents, AI inventory
3.2.2 MediAssist at Level 1
The section shows how Bluebonnet turned its AI charter into limits and approval requirements for MediAssist, with added safeguards for medication support.
- Charter rule and authority: Explain the charter’s rule for clinical AI and the RAI Office’s authority over proposed uses.
Why it matters: The charter gives teams a boundary for evaluating proposals before patient-facing features launch. - Patient-facing product limits and approvals: Describe what MediAssist’s acceptable use policy permitted and prohibited, and the two decisions required before work began on features that directly discussed health concerns with patients.
Why it matters: These boundaries and approvals keep patient-facing features within an agreed clinical purpose. - Medication support safeguards: Explain why the medication-support proposal required additional safeguards and name those safeguards.
Why it matters: The example shows how a proposal can call for safeguards beyond the standard approval route.
Key terms: charter, clinical judgment, RAI Office, senior approval, acceptable use policy, Steering Committee, clinical purpose and limits, medication support, independent clinician validation, published treatment guidelines
3.3 Risk & Impact Assessment (Level 2)
Level 2 uses a feature-specific AI Impact Assessment to uncover risks that general policy may miss and to decide what must be addressed before a proposal moves forward.
- Feature-Specific Risk Discovery: Explain why a feature that appears to fit policy still needs an AI Impact Assessment, using Derm-Cam to illustrate risks revealed by concrete questions.
Why it matters: Asking feature-specific questions can expose problems that policy compliance and a vendor contract leave unresolved. - Conditional Assessment Decision: Describe the decision an AI Impact Assessment should produce and how conditions and evidence shape whether Derm-Cam can move forward.
Why it matters: A conditional decision turns discovered risks into requirements while changes are still cheaper to make.
Key terms: Level 2, AI Impact Assessment, Derm-Cam, vendor review, decision and conditions, required evidence, clinician fallback
3.3.1 Key artifacts
This section explains the artifacts that make an AI Impact Assessment a consistent, actionable review of a GenAI proposal and connect its findings to later work.
- AIIA scope and questions: Explain what an AIIA records and why its questions must be answered early enough to shape the design. Describe the kinds of questions it asks about affected people, data, model evaluation, risks, controls, post-launch responsibilities, and external tools.
Why it matters: An early, concrete assessment can change the system’s design rather than merely document risks after the fact. - AIIA and conformity assessment: Distinguish a well-structured AIIA from an EU AI Act conformity assessment for a high-risk AI system.
Why it matters: A useful internal review should not be mistaken for completion of a separate legal requirement. - Actionable assessment findings: Describe how AIIA findings become concrete requirements or obligations, and how each finding is tracked.
Why it matters: Findings can guide delivery and be audited only when they are specific, assigned, and traceable. - Parallel third-party assessment: Explain when a third-party risk assessment runs alongside the AIIA and what decision its checks support.
Why it matters: A proposal’s risks include dependencies on vendors, not just the team’s own design. - Risk-triggered follow-up reviews: Describe how elevated risks found by the AIIA can trigger additional reviews and how their results stay connected to the proposal.
Why it matters: The assessment is a way to route specific risks to the right further review, rather than treating every proposal identically. - Shared assessment documentation: Describe how shared artifacts, early system documentation, and maintained guidance help teams conduct consistent assessments.
Why it matters: Shared guidance and developing documentation make assessments repeatable across teams and useful at later review stages.
Key terms: AI Impact Assessment, AIIA, design constraints, conformity assessment, EU AI Act, high risk, AIIA findings, residual risk, owner, due date
3.3.2 Key Metrics
This section explains why an AI Impact Assessment process needs metrics beyond completed templates to show whether reviews and follow-ups work.
- Assessment Process Effectiveness: Explain why routing every proposal through the AIIA template is not enough, and describe what the RAI Office and Steering Committee need to track to see whether the review process works.
Why it matters: These measures help reveal when a review process looks complete on paper but is failing in practice.
Key terms: AIIA template, RAI Office, Steering Committee, AI Impact Assessment
3.3.3 MediAssist at Level 2
Derm-Cam’s AI Impact Assessment turns privacy, retention, and fairness concerns into assigned, verifiable conditions for approval.
- Derm-Cam assessment findings: Describe the three concerns identified for Derm-Cam: identifiable skin photos that may be health data, vendor retention exceeding policy, and untested performance across skin tones.
Why it matters: Specific findings make it possible to assign and check the work needed before launch. - Assigned risk mitigation work: Explain how the assessment assigned responses to the findings, including legal review, changed retention terms, skin-tone testing, clinician fallback, and approved patient-facing messages.
Why it matters: An assessment becomes useful when its findings lead to concrete work and limits on what the feature can tell patients. - Traceable conditional approval: Explain how MediAssist made findings verifiable and why Derm-Cam received only conditional approval.
Why it matters: Conditional approval depends on evidence that required changes happened, not just promises to make them.
Key terms: AI Impact Assessment, health data, data retention, skin tones, DPIA, consent flow, fallback logic, patient-facing messages, vendor deletion records, storage logs
3.4 Design & Implementation Reviews (Level 3)
Level 3 reviews check an AI system’s design before it is built, turn risks and requirements into workable controls, and track decisions through release.
- Early architecture risk review: Explain why design and implementation reviews start before the system is built, using Derm-Cam’s direct vendor API call as an example of a convenient choice that may introduce risk.
Why it matters: An early review can prevent a quick architectural choice from becoming an expensive risk to fix. - Practical control requirements: Explain how a Level 3 review should handle requirements that are sound in principle but infeasible or disproportionately costly in the actual system.
Why it matters: Controls only reduce risk when they can be implemented responsibly in the real system. - Incomplete control coverage: Describe how a review handles a control that addresses only part of its intended risk.
Why it matters: A control’s presence is not proof that it covers every place the risk appears. - Vendor control responsibilities: Explain why implementation reviews clarify which AI controls belong to the vendor and which belong to the customer.
Why it matters: Unclear ownership can leave an important safeguard undone by both parties. - Review findings and release tracking: Describe how review decisions and Level 2 requirements are recorded and carried into engineering work, and distinguish those release records from the AI inventory.
Why it matters: A requirement cannot reliably affect the shipped system unless someone owns and tracks the work.
Key terms: Design & Implementation reviews, design review, technical standard, Technical Feasibility, L3 review, L2 policy, risk approver, control, compensating control, residual risk
3.4.1 Key Artifacts
This section explains how implementation reviews connect assessment conditions to concrete design evidence and use flexible technical standards for consistent review.
- AIIA conditions in implementation artifacts: Trace an AIIA condition to the design, backlog work, and test that will implement and prove it before launch, and describe how reviewers check the requirements they own.
Why it matters: Linking conditions to implementation evidence lets reviewers see whether assessment requirements will actually be met. - Consistent yet flexible technical standards: Explain how technical standards make implementation reviews consistent and why those standards must accommodate technical infeasibility.
Why it matters: Standards prevent inconsistent reviews without demanding controls a deployment cannot technically support.
Key terms: AIIA condition, design or backlog, implementation review, technical standards, approved gateway, gateway routing, technical infeasibility
3.4.2 Key metrics
This subsection explains what Level 3 implementation-review metrics should reveal and where organizations can find the records to support them.
- Implementation Review Metric Purpose: Explain that Level 3 metrics should show whether technical reviews happen early enough and whether their findings become work, using records organizations already have.
Why it matters: These metrics help distinguish a timely, actionable review from one that merely takes place.
Key terms: Level 3 review metrics, technical review, architecture review tickets
3.4.3 MediAssist at Level 3
Derm-Cam’s Level 3 architecture review traces the proposed build, identifies privacy and security risks, and turns assessment conditions and findings into owned prelaunch work.
- Architecture Trace to Build Controls: Explain how tracing Derm-Cam’s flow showed where its AIIA conditions had to be implemented, including retention, clinician review, and patient-facing wording.
Why it matters: Tracing the real flow makes governance conditions concrete enough for engineers to implement. - Photo Identifier Privacy Controls: Describe how uploaded photos could collect unintended personal data and how Derm-Cam changed the upload, processing, and logging flow to limit it.
Why it matters: A photo can expose personal information even when the product asks only for an image of a mole. - Image Text Injection Boundary: Trace how OCR could turn text hidden in an uploaded image into an instruction to the LLM, and explain Derm-Cam’s resulting response constraints.
Why it matters: Removing an unnecessary input path and constraining outputs helps keep a medical feature within its approved purpose. - Owned Prelaunch Remediation Work: Describe how the architecture review findings became assigned engineering tasks, including how proposed exceptions were handled.
Why it matters: Named owners, deadlines, and exception approval keep review findings from remaining unactioned observations.
Key terms: architecture review, AIIA conditions, retention limit, clinician review, approved wording, EXIF metadata, identifiers, upload flow, approved retention rule, OCR
3.5 Launch Readiness Review (Level 4)
Launch Readiness Review checks that required controls work in the version being released, scales approval to risk, and prevents launch when required checks remain unresolved.
- Implemented Control Verification: Explain why Launch Readiness Review verifies that required controls are working, rather than accepting implementation tickets as proof. Describe the kinds of checks illustrated by Derm-Cam, including image metadata removal, approved patient wording, required reviews and tests, retention settings, and whether the tested build and configuration match the planned release.
Why it matters: A completed task is not evidence that the released system has the required safeguards. - Risk Scaled Review Approval: Distinguish how the review and approval of LRR can differ between low-risk and high-risk systems.
Why it matters: The approval process should reflect the system’s risk rather than impose the same sign-off on every release. - Failed Check Launch Gate: State the permitted responses when a required LRR check fails and explain when the feature must not launch.
Why it matters: A failed check must lead to an explicit risk decision, not be silently carried into production. - Release Candidate Evidence Quality: Explain why a generic completed test or vendor claim may be inadequate LRR evidence, and describe the evidence Derm-Cam needs for adversarial and skin-tone performance checks.
Why it matters: Evidence is useful for launch only if it tests the relevant failure mode on the system that will actually ship.
Key terms: Launch Readiness Review (LRR), implementation review, approved patient wording, release, LRR, low risk system, high risk systems, AIIA, final approval, required check
3.5.1 Key artifacts
This subsection identifies the small evidence set needed for a launch readiness review.
- Launch review evidence set: State what the product owner and required approvers need to know for a launch readiness review: what was tested, which version, what failed, what was fixed, and who approved the launch.
Why it matters: A compact evidence set lets approvers see the basis for the launch decision.
Key terms: launch readiness review, evidence set, product owner, required approvers
3.5.2 Key Metrics
This subsection states what launch readiness metrics check before release.
- Launch readiness metric checks: Explain what the launch readiness metrics check: timely required reviews, appropriate evidence, and the handling of failed tests.
Why it matters: These checks show whether launch decisions have the required review, evidence, and treatment of failed tests.
Key terms: launch reviews, evidence, failed tests, risk owner
3.5.3 MediAssist at Level 4
This subsection shows how Derm-Cam’s launch review used evidence for the exact release candidate, risk-appropriate reviewers, and a final approval gate.
- Release Candidate Approval Gate: Explain how Derm-Cam’s launch decision was tied to the exact release candidate and the conditions set by earlier reviews. Describe what the release record contained and when the steering committee approved rollout.
Why it matters: Approval applies to an evidenced release candidate, not a general assurance that the product is ready. - Risk Appropriate Launch Reviewers: Distinguish the lighter approval path described for a low-risk chatbot from Derm-Cam’s named-reviewer approach, and describe what each reviewer checked.
Why it matters: The review path reflects the kinds of harm the release could cause.
Key terms: launch review, release candidate, release record, must-fix finding, controlled rollout, product owner’s declaration, named reviewers, build controls, adversarial image tests, clinician fallback
3.6 Operations & Monitoring (Level 5)
Level 5 keeps a live AI system under control through records, monitoring, change review, incident recovery, and secure retirement.
- Need for post-launch governance: Explain why launch approval alone cannot control a live AI application: its models, settings, connections, and user requests can change after testing.
Why it matters: A successful launch review is a starting point, not a permanent assurance. - Configuration and request records: Distinguish the system configuration record from the request record and explain how each supports investigation without retaining unnecessary personal data.
Why it matters: Investigators need to reconstruct both the setup in force and what occurred in a particular request. - Multi-stakeholder monitoring roles: Describe how engineering, security, customer support, and, in high-stakes applications, domain experts monitor different signs of trouble.
Why it matters: Combining these perspectives makes problems visible that technical error monitoring alone could miss. - Baseline-based change review: Explain how significant model, prompt, data-source, service-connection, or safety-setting changes should be evaluated before users receive them.
Why it matters: Reviewing changes against the approved system helps catch regressions before they affect users. - Incident containment and recovery: Trace the immediate response to a serious live-system failure and the checks required before the system resumes.
Why it matters: A fix is not enough unless the team limits harm during the incident and verifies that resumption is justified. - Secure application decommissioning: Describe the obligations involved in retiring an AI application rather than simply abandoning it.
Why it matters: Taking the product out of sight does not remove its remaining data, access, or unresolved obligations.
Key terms: Level 5, post launch, operational discipline, System Configuration Record, Request Record, Multi-stakeholder Monitoring, domain experts, Change Review, known baseline, material decline
3.6.1 Key artifacts
The section explains how to make operational evidence findable after a failure and what agentic-system logs must record.
- Findable failure evidence: Explain why operations and monitoring records need an assigned owner and location.
Why it matters: Investigators need to know where to find the records that explain a failure. - Agentic action log contents: Describe what an agentic-system log should capture beyond the final prompt and answer.
Why it matters: A final answer alone does not show the agent’s tool use, access, requested action, or human approval.
Key terms: application logs, security alerts, support tickets, change records, owner, location, agentic systems, tool, action requested, approved
3.6.2 Key Metrics
3.6.3 MediAssist at Level 5
A Derm-Cam pilot incident shows how a missed alert and incomplete records hinder investigation, and how the team contained the risk and documented what to capture next time.
- Uncaught pilot safety breach: Describe how Derm-Cam violated its approved rules during the pilot and how the problem was discovered.
Why it matters: The case shows why sampled review matters even when an alerting system is in place. - Missing investigation evidence: Explain why the available record could not establish what had changed or which safeguard failed.
Why it matters: Without a trace of what produced an output, investigators cannot pinpoint the failed safeguard. - Pilot containment and incident record: Describe the team's immediate response and what it specified for future investigations.
Why it matters: The response limits further exposure while making the failure traceable and assignable.
Key terms: Derm-Cam, clinician review, alert, prompt, provider model, safeguard, incident, logs, approved wording
3.7 Learning & Improvement (Level 6)
Level 6 turns incidents and other findings into changes to future releases, while preserving the evidence and decisions needed to investigate problems or limit a product.
- Findings into lasting controls: Explain why a finding should change the failed checklist, template, test, monitoring rule, or product limit rather than remain only in an incident ticket.
Why it matters: A recorded incident prevents recurrence only when it changes how the product is governed. - Bad-answer investigation records: Describe what future Derm-Cam releases had to record so the team could investigate a bad answer.
Why it matters: Without a record of the exchange, the team may be unable to investigate what went wrong. - Limits and closure decisions: Explain when learning may require limiting, pausing, or retiring a product, and what the decision record should capture.
Why it matters: Some findings call for a decision about whether the product can responsibly continue, not just another fix.
Key terms: finding, incident ticket, checklist, monitoring rule, product limit, Derm-Cam, model or vendor endpoint, prompt, clinician review, conditions approved at launch
3.7.1 Key Artifacts
This section explains how Level 6 turns findings into changes to governing artifacts and links records so reviewers can check whether fixes worked.
- Findings That Change Artifacts: Explain why logging incidents and holding retrospectives are not enough: a finding produces improvement only when it changes the artifacts that govern system behavior.
Why it matters: It distinguishes documenting a problem from changing how the system is governed. - Traceable Fix and Test Records: Describe how linked records let a reviewer trace an incident through its fix and a changed test, then check whether the failure returned.
Why it matters: A traceable chain makes it possible to check whether a response to an incident held up in practice.
Key terms: finding, artifacts, closed finding, records, incident, fix, changed test
3.7.2 Key Metrics
This section identifies the metrics used to check whether Level 6 findings lead to lasting improvements.
- Level 6 follow-through metrics: Explain how Level 6 measures follow-through on a finding: whether the owner closed the fix on time, whether the issue returned, and whether the finding changed the control that missed it.
Why it matters: These measures show whether a finding produced a lasting improvement rather than just a completed fix.
Key terms: Level 6 metrics, follow through, finding
3.8 Governing Agents at Runtime
This section explains how to extend lifecycle governance to agents by controlling actions at runtime, tracing delegated authority, governing memory, and finding and stopping active agents.
- Contextual Action Policies: Explain why an agent's actions need conditions based on its live state rather than blanket tool permissions, and describe how rules can be checked before each action.
Why it matters: The same tool call can be safe or unsafe depending on what happened earlier in the run. - Delegation Authority Trails: Explain why sequences of actions and multi-hop delegation require both a complete requester trail and narrowing authority at each handoff.
Why it matters: A record of the last agent alone cannot establish accountability or prevent privilege escalation through delegation. - Governance of Agent Memory: Describe the privacy obligations created when agent memory persists across runs and how MediAssist's memory design supports isolation and deletion.
Why it matters: If memory is scattered or unlabelled, an organization may be unable to find, isolate, or delete a patient's retained information. - Runtime Control Capabilities: Distinguish pre-action policy enforcement, identity brokering, and logging as capabilities needed to govern agents, regardless of vendor.
Why it matters: Separating these capabilities helps a team assess controls even as agent-governance products change. - Agent Inventory and Visibility: Describe how to establish an agent registry and find active agents that did not enter through formal registration, then observe whether they use the control plane.
Why it matters: Controls cannot reliably protect agents the organization has not found or that bypass their enforcement path. - Agent Stop Controls: Describe basic ways to stop a misbehaving agent and the automatic limits that can halt it without waiting for human detection.
Why it matters: A team needs a way to halt harmful behavior promptly, not merely investigate it afterward.
Key terms: live state, condition, policy rules, AI Gateway, multi-hop delegation, full chain of requesters, permission narrowing, privilege escalation, agent memory, retention periods
3.9 Summary
The summary shows how the Six-Level Model turns GenAI policies into controls that can be tested, monitored, and improved in practice.
Where this is used later
- (chapter intro) (chapter 7) · The governance framework introduced earlier in the book.
- Finding yourself on the map (chapter 7) · The governance framework whose basics teams should cover while resolving uncertainty.
- What can you govern today (chapter 8) · The governance framework for making decisions about a real system
- (chapter intro) (chapter 6) · Ethical standards begin at Level 1 Strategy in the six-level governance framework.
- Transparency and AI Literacy Obligations (chapter 7) · The organizational AI inventory used to identify tools in use.
- What high-risk providers must do (chapter 7) · The distinction matters when considering high-risk provider duties.
- Classifying Your System (chapter 7) · System classification informs which regulatory follow-ups may apply.
- Overcollection (chapter 5) · The unintended identifiers illustrate overcollection of personal data.
Concepts explained in this chapter
- policyA set of conditions governing an agent’s actions according to its state and context.
- agent registryAn agent registry is a maintained record of active agents and their purposes, owners, and operational exposure.
- ai gatewayAn AI gateway is an intermediary layer that applies governance, access, and operational controls to requests between AI systems and services.
- delegationDelegation is the transfer of work or authority from one agent or actor to another.
- identity brokerAn identity broker mediates identities and credentials between agents, people, and services to establish accountability and access.
- kill switchA kill switch is a control that can rapidly disable an AI system or model to limit harm or enable investigation.
- memoryAgent memory is information retained by an agent for use in later interactions or tasks.
- policy engineA policy engine is a service or software component that evaluates requests against defined policies and returns decisions such as allow or deny.
- 6L-G modelThe book’s six-level framework for translating generative AI governance principles into organizational and technical controls.
- impact assessmentAn impact assessment is a documented evaluation of the risks an AI system introduces for individuals and society, recorded here as a structured review of a GenAI proposal.
- architecture reviewAn implementation review is a formal examination of an AI system design to verify that required legal, security, privacy, and operational controls are feasible and addressed.
- acceptable use policyAn acceptable use policy is a formal set of rules specifying permitted and prohibited uses of organizational systems, tools, and data.
- third-party risk assessmentA third-party risk assessment evaluates whether an external provider and its services meet the security, privacy, operational, and contractual requirements of a proposed use.
- incident responseIncident response is the organized process for detecting, containing, investigating, remediating, and documenting harmful or unauthorized system events.
- AI governanceAI governance is the system of principles, processes, and oversight used to direct and control the development and use of artificial intelligence.
- red-teamingAdversarial evaluation in which authorized testers simulate attackers to discover weaknesses, failures, or harmful behaviors.
- data retentionData retention is the storage of data for a defined period under operational, legal, or policy requirements.
- fairnessFairness is a normative criterion for evaluating whether an AI system treats people or groups equitably, with distinct formal definitions that may conflict.
- fallbackFallback logic is a programmed alternative behavior invoked when a primary operation fails, is unavailable, or cannot safely proceed.
- logLogging is the recording of system events and associated information for monitoring, audit, or analysis.
- strategy and policy levelThe 6L-G governance level that establishes organizational direction, AI policies, leadership, accountability, and oversight.
- learning and improvementThe Learning & Improvement level of the 6L-G model uses feedback and flaws found in live systems to improve AI governance, designs, processes, tests, and controls.
- human-in-the-loopAn approach in which a person participates in an AI system’s operation, such as by reviewing or authorizing its outputs or actions.
- driftA change in a model’s behavior over time caused by model updates or changes in its inputs.
- threat modelA structured technique for identifying, analyzing, and prioritizing threats to a system.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
