Chapter 9: Governance and operations
Introduces governance and operations as the assurance disciplines that help context-aware AI systems run reliably and responsibly in production.
After this chapter you will be able to
- Governance and operations roles. Explain why governance and operations matter for production context-aware AI systems: they provide organizational policies, security guardrails, human oversight workflows, and operational pipelines for reliable and responsible use.
- AI governance scope and purpose. Explain how organizational structures, processes, and guardrails govern the design and operation of AI systems, and why they are used.
- Six governance pillars. State the six pillars adopted by the book and explain why weakness in one matters to the whole AI system.
- Permission propagation through context pipelines. Explain why context pipelines must carry user identity and access permissions into retrieval, memory reads, and tool calls.
- Task and access boundaries. Describe how data minimization, purpose limitation, and scoped authorization constrain what enters runtime context.
- Sensitive data handling and retention. Describe how redaction, synthetic test data, and retention limits reduce exposure at different points in the data lifecycle.
- Assembly logging and provider safeguards. Explain where privacy controls apply in a support assistant and why traces and external providers need separate safeguards.
- Prompt injection and trust boundaries. Explain how prompt injection turns untrusted context data into apparent instructions, why the model is susceptible, and how direct and indirect injection differ.
- Layered defenses beyond prompts. Justify why defensive prompt wording alone cannot secure a context-aware system and describe the layered controls this section proposes.
- Accountable AI decision traces. Explain why accountability in context engineering requires traceability and describe what must be recorded to reconstruct a disputed customer support answer or refund recommendation.
- Two Audiences for Transparency. Explain what transparency means in an AI system, how it relates to explainability, and what information it provides to users and operators. Illustrate it with the customer support assistant.
- Fairness in comparable outcomes. Explain that fairness seeks to prevent unjustified differences in outcomes between people or groups, especially differences tied to protected characteristics. Describe what comparable treatment means for similar customer support issues.
- Fairness across context sources. Explain how the context management layer can address fairness through retrieval, instructions, and checks of intermediate context. Describe why controls should examine context sources and final outputs, using the customer-memory example.
- Risk-based oversight levels. Distinguish human-in-the-loop, human-on-the-loop, and human-out-of-the-loop oversight, and explain how risk, impact, and reversibility guide the choice.
- Customer support escalation controls. Describe when the customer support assistant needs human oversight and what operational controls make that oversight possible.
- Verifiable governance controls. Explain why governance pillars must become concrete, verifiable controls that teams can check before releasing a context-engineered system.
- Human judgment in AI systems. Explain why high-reliability AI systems combine automation with human judgment and what human operators contribute as context injectors and fail-safes.
- Risk-based review pipeline. Trace how a proposed agent action is routed through policy checks, possible human review, and recording of the final decision.
- Coupled operational concerns. Explain why operating a production context-aware system involves more than keeping its infrastructure running: changes to context can affect both model behavior and infrastructure load, linking reliability, cost, latency, and other operational concerns.
- DevOps extension for LLMs. Distinguish the DevOps foundations named here from the additional operational challenge posed by LLM-based systems, and explain why context-aware architectures need specialized operational disciplines.
- Adoption boundaries and responsibilities. Explain how an adoption model sets an organization’s operating boundary for context artifacts, data flows, model access, and serving responsibility. Describe why one organization might use different postures for different applications.
- Vendor products versus model APIs. Distinguish the SaaS consumer and API integrator postures by what the provider runs, what context engineering the organization controls, and what responsibilities remain with the organization.
- Self-managed model postures. Distinguish LLM off-the-shelf deployment from the model hoster posture in terms of model and infrastructure control, serving obligations, and governance integration.
- DevOps and model hosting operations. Explain why LLM-based systems call for specialized extensions of DevOps and describe the model-hosting progression from MLOps through FMOps to LLMOps.
- Model consumer operations branch. Distinguish the model-consumer branch from model-hosting operations and explain how GenAIOps leads to ContextOps.
- ContextOps responsibilities and ownership. Describe the responsibilities assigned here to PromptOps, RAGOps, and AgentOps, and explain why their labels and team ownership should not be treated as fixed.
- LLMOps scope and lifecycle. Explain when an organization needs the broader LLMOps lifecycle and why operating an LLM differs from operating a traditional predictive model.
- Model customization intervention layers. Distinguish the customization approaches by what each changes or learns, and describe their operational implications.
- Serving-side cache responsibilities. Explain which cached-prefix decisions belong to LLMOps rather than the application, and why cache reuse requires isolation.
- Production model management controls. Describe the controls used to sustain reliability, safety, and traceability after a model reaches production.
- Prompts as behavioral configuration. Explain why PromptOps treats system and user prompt templates as configuration artifacts that need management similar to source code.
- Production prompt change practices. Describe how versioning, automated evaluation, A/B testing, and monitoring support prompt changes in production.
- RAGOps purpose and workflows. Explain why RAGOps is needed in production and distinguish its two interconnected workflows: runtime query processing and data-management indexing.
- Runtime query workflow and feedback. Trace the operations on the runtime query path and describe how telemetry feeds regression traces into automated evaluation.
- Knowledge store update lifecycle. Describe the five data-management stages and how they check and release updates to RAG knowledge stores.
- AgentOps operational concerns. Explain why agentic systems need dedicated operations, with state management and traceability as central concerns.
- Agent monitoring and checkpoints. Describe what AgentOps monitoring captures beyond traditional system signals and why it checkpoints memory and environment state.
- Agent failure investigation lifecycle. Trace how anomaly detection, root cause analysis, and iterative resolution follow monitoring in the four-stage AgentOps lifecycle.
- Governed external tool actions. Explain why tool use requires explicit governance and describe the controls for tools and consequential actions.
- Context as code scope. Explain what it means to manage context configurations as first-class software artifacts and identify the kinds of context sources this applies to.
- Four operational practices. Describe the four operational practices of context as code and what each contributes.
- Scope and risk of deployment. Explain why deploying a context-engineered AI system includes more than application code and why even small context changes require care.
- Controlled rollout strategies. Distinguish blue-green, canary, and shadow deployment by how each handles traffic, user exposure, and risk.
- Release metrics and rollback triggers. Describe how explicit metrics, monitoring windows, and rollback triggers govern whether a rollout continues, expands, pauses, or reverses.
- Presidio redaction flow. Trace how the example uses Presidio to identify and redact sensitive information in input text.
- Redaction quality tradeoffs. Explain what affects redaction quality and why different workflows may need different safeguards.
- Schema and policy validation. Explain how validating an agent’s output against a JSON Schema derived from a Pydantic model supports runtime safety checks, and describe the required output fields and policy decisions in this example.
- Group-level fairness comparison. Explain how a fairness audit compares recommendations across sensitive groups, using selection rates for young and senior credit applicants as the example.
- Four-fifths rule interpretation. Describe what a demographic parity ratio below 0.80 signals under the four-fifths rule and why that signal is not a final fairness judgment.
- Story-writing approval loop. Trace how the story-writing assistant uses human input: the user supplies an initial premise, the AI generates a paragraph, and the application pauses for the user to approve it or give plot-changing instructions before the narrative proceeds.
- Unified AI gateway interface. Explain how an AI gateway lets an application work with multiple LLM providers through one interface while centralizing routing, retries, and provider selection.
- Mocked provider fallback pattern. Trace the example's primary-model request and simulated failure through fallback to an alternate provider, and distinguish its mocked responses from production uses of the same architecture.
- Versioned prompt evaluation gate. Describe how the support-ticket prompt is kept in a separate, versioned Jinja2 file and checked for classification regressions. Explain that a Python evaluator renders it against test tickets with expected classifications and that GitHub Actions runs the same check in CI.
What it builds on
- Evaluation (chapter 8) · Introduces evaluation, one activity that governance can set policies for.
- Observability (chapter 8) · Introduces observability, which informs governance expectations for monitoring.
- Retrieval-augmented generation (chapter 3) · Introduces retrieval as a source of context.
- Tools (chapter 4) · Introduces tools as an agent context source.
- Memory (chapter 5) · Introduces memory as an agent context source.
- Bias, safety, and privacy (chapter 1) · the earlier discussion of bias, safety, and privacy as LLM limitations
- Taxonomy for context sources (chapter 1) · the distinction among sources of context supplied to an LLM
- Best practices for prompt design (chapter 6) · defensive prompt design, which this section says can reduce misinterpretation but cannot enforce security boundaries
Section by section
Introduction
Introduces governance and operations as the assurance disciplines that help context-aware AI systems run reliably and responsibly in production.
- Governance and operations roles: Explain why governance and operations matter for production context-aware AI systems: they provide organizational policies, security guardrails, human oversight workflows, and operational pipelines for reliable and responsible use.
Why it matters: This frames the purpose of the governance and operations practices discussed in the chapter.
Key terms: governance, operations, cross-cutting assurance disciplines
9.1 Governance
Defines AI governance, identifies complementary governance frameworks and six shared pillars, and explains why context pipelines must enforce access permissions.
- AI governance scope and purpose: Explain how organizational structures, processes, and guardrails govern the design and operation of AI systems, and why they are used.
Why it matters: Governance applies to decisions throughout an AI system’s lifecycle, not just to a final compliance check. - Complementary governance initiatives: Distinguish the purposes served by the governance initiatives named in this section rather than treating them as interchangeable definitions.
Why it matters: Different governance resources address different needs, so an organization may use more than one. - Six governance pillars: State the six pillars adopted by the book and explain why weakness in one matters to the whole AI system.
Why it matters: The pillars provide a common way to consider governance without reducing it to a single concern. - Permission propagation through context pipelines: Explain why context pipelines must carry user identity and access permissions into retrieval, memory reads, and tool calls.
Why it matters: Access checks must follow the user into each source of context to prevent unauthorized disclosures.
Key terms: AI governance, structures, processes, guardrails, OECD AI Principles, NIST AI Risk Management Framework, WEF AI Global Alliance, ISO/IEC 42001, EU AI Act, governance pillars
9.1.1 Privacy
Explains where privacy risks arise in LLM systems and how to limit sensitive-data exposure across context assembly, testing, retention, logging, and provider use.
- Privacy scope and exposure risks: Explain what privacy governs in an LLM system and distinguish training-data memorization from runtime context exposure.
Why it matters: It separates a model-training risk from the privacy risks an application can manage when assembling context. - Task and access boundaries: Describe how data minimization, purpose limitation, and scoped authorization constrain what enters runtime context.
Why it matters: These checks prevent unnecessary or unauthorized information from entering a prompt in the first place. - Sensitive data handling and retention: Describe how redaction, synthetic test data, and retention limits reduce exposure at different points in the data lifecycle.
Why it matters: Privacy controls must address data before use, during testing, and after its operational need ends. - Assembly logging and provider safeguards: Explain where privacy controls apply in a support assistant and why traces and external providers need separate safeguards.
Why it matters: Protecting the prompt alone is insufficient if logs or an external provider retain the same sensitive context.
Key terms: privacy, personally identifiable information (PII), training-data memorization, runtime context exposure, data minimization, purpose limitation, scoped authorization, automated redaction, synthetic data generation, explicit retention limits
9.1.2 Safety and security
This section distinguishes AI safety from security, explains why prompt injection threatens both, and describes layered controls for context-aware systems.
- Safety and security distinction: Distinguish AI security, which protects context sources and systems from manipulation or exploitation, from AI safety, which prevents harmful, toxic, or misleading outputs.
Why it matters: The distinction helps identify whether a control protects the system, its users, or both. - Prompt injection and trust boundaries: Explain how prompt injection turns untrusted context data into apparent instructions, why the model is susceptible, and how direct and indirect injection differ.
Why it matters: Recognizing where an instruction came from is central to spotting attempts to hijack an agent. - Layered defenses beyond prompts: Justify why defensive prompt wording alone cannot secure a context-aware system and describe the layered controls this section proposes.
Why it matters: Layered controls limit harm even when an attacker gets malicious text into the model's context.
Key terms: AI security, AI safety, guardrails, prompt injection, direct injection, indirect injection, untrusted context data, defense in depth, input and output guardrails, provenance labeling
9.1.3 Accountability
Explains how traceable, versioned records make organizations accountable for AI decisions and allow disputed outputs to be reconstructed.
- Accountable AI decision traces: Explain why accountability in context engineering requires traceability and describe what must be recorded to reconstruct a disputed customer support answer or refund recommendation.
Why it matters: A disputed AI decision can be investigated only if its context, actions, and approvals can be traced.
Key terms: accountability, full traceability, version control, immutable audit traces, trace record
9.1.4 Transparency
Defines transparency in AI systems and shows how it serves users and operators.
- Two Audiences for Transparency: Explain what transparency means in an AI system, how it relates to explainability, and what information it provides to users and operators. Illustrate it with the customer support assistant.
Why it matters: Transparency gives users clearer grounds for trusting a response and gives operators records they can inspect.
Key terms: transparency, explainability, user-facing transparency, operator-facing transparency, execution traces, context lineage
9.1.5 Fairness
Defines fairness in AI outcomes and explains why fairness checks must examine the context that shapes those outcomes, not just final responses.
- Fairness in comparable outcomes: Explain that fairness seeks to prevent unjustified differences in outcomes between people or groups, especially differences tied to protected characteristics. Describe what comparable treatment means for similar customer support issues.
Why it matters: Comparable cases provide a concrete way to spot potentially unfair treatment. - Fairness across context sources: Explain how the context management layer can address fairness through retrieval, instructions, and checks of intermediate context. Describe why controls should examine context sources and final outputs, using the customer-memory example.
Why it matters: Checking only the answer can miss biased information that shaped how the assistant treated a customer.
Key terms: fairness pillar, unjustified disparities, protected characteristics, context management layer, biased retrieval, discriminatory heuristics, intermediate context representations, memory entries
9.1.6 Human oversight
This section explains how to match human oversight to an AI action’s risk and how to make oversight actionable in a customer support assistant.
- Risk-based oversight levels: Distinguish human-in-the-loop, human-on-the-loop, and human-out-of-the-loop oversight, and explain how risk, impact, and reversibility guide the choice.
Why it matters: The level of human control should fit the consequences of the action rather than be the same for every task. - Customer support escalation controls: Describe when the customer support assistant needs human oversight and what operational controls make that oversight possible.
Why it matters: Human oversight only works in practice when people can receive, review, and account for cases that need their judgment.
Key terms: human oversight, human-in-the-loop (HITL), human-on-the-loop (HOTL), human-out-of-the-loop (HOOTL), risk, impact, escalation routes, review queues, operator interfaces
9.1.7 Governance in practice
This subsection turns governance pillars into production-readiness checks and notes that some checks require human judgment.
- Verifiable governance controls: Explain why governance pillars must become concrete, verifiable controls that teams can check before releasing a context-engineered system.
Why it matters: Release checks make governance requirements something a team can verify rather than merely endorse. - Limits of technical guardrails: Explain why some governance checks cannot be satisfied by technical guardrails alone.
Why it matters: A release process must account for judgments that cannot be reduced to automated safeguards.
Key terms: pillar, concrete control, production readiness gate, human judgment, technical guardrails
9.2 Human-in-the-loop
Explains how production AI systems use human-in-the-loop review to handle risky or ambiguous decisions while managing the cost of oversight.
- Human judgment in AI systems: Explain why high-reliability AI systems combine automation with human judgment and what human operators contribute as context injectors and fail-safes.
Why it matters: Human review gives an automated system a way to resolve decisions it should not make alone. - Risk-based review pipeline: Trace how a proposed agent action is routed through policy checks, possible human review, and recording of the final decision.
Why it matters: Selective routing preserves automation while making consequential decisions reviewable and traceable. - Operational roles of review: Describe the four kinds of intervention HITL provides: output validation, ambiguity resolution, approval gates, and dynamic context enrichment.
Why it matters: Different failure modes call for different contributions from a human reviewer. - Costs of human escalation: Explain why production HITL systems limit escalation and name the operational measures used to focus reviewer attention.
Why it matters: A review process must protect safety without becoming a bottleneck for the whole application.
Key terms: HITL, dynamic context injectors, authoritative fail-safes, policy engine, confidence and risk thresholds, review queue, trace store, output validation, ambiguity resolution, safety and approval gates
9.3 Operations
Introduces why production context-aware systems need operations that account for both changing context and the distinctive behavior of LLMs.
- Coupled operational concerns: Explain why operating a production context-aware system involves more than keeping its infrastructure running: changes to context can affect both model behavior and infrastructure load, linking reliability, cost, latency, and other operational concerns.
Why it matters: A context change can have operational consequences as well as change an answer. - DevOps extension for LLMs: Distinguish the DevOps foundations named here from the additional operational challenge posed by LLM-based systems, and explain why context-aware architectures need specialized operational disciplines.
Why it matters: Conventional release and infrastructure practices alone do not address all the variability of LLM-based systems.
Key terms: operations, context-aware systems, infrastructure load, DevOps, nondeterminism, probabilistic outputs, evolving data dependencies
9.3.1 Adoption models
This section compares four ways organizations adopt generative AI by the control, trust boundaries, and operational responsibilities each gives them.
- Adoption boundaries and responsibilities: Explain how an adoption model sets an organization’s operating boundary for context artifacts, data flows, model access, and serving responsibility. Describe why one organization might use different postures for different applications.
Why it matters: The posture you choose sets the boundary for both context engineering and operational ownership. - Vendor products versus model APIs: Distinguish the SaaS consumer and API integrator postures by what the provider runs, what context engineering the organization controls, and what responsibilities remain with the organization.
Why it matters: Using a hosted model API gives you much more control over context than using a finished SaaS product, but it does not remove your application-side duties. - Self-managed model postures: Distinguish LLM off-the-shelf deployment from the model hoster posture in terms of model and infrastructure control, serving obligations, and governance integration.
Why it matters: Running a model yourself changes the work from assembling context alone to operating serving infrastructure and, at the hoster level, integrating it with enterprise governance.
Key terms: adoption model, adoption posture, trust boundary, context artifacts, SaaS consumer, API integrator, LLM as a service, context stack, LLM off-the-shelf, model hoster
9.3.2 DevOps for AI systems
This section explains how DevOps branches into operational disciplines for model hosts and model consumers, with ContextOps organizing responsibilities around context sources.
- DevOps and model hosting operations: Explain why LLM-based systems call for specialized extensions of DevOps and describe the model-hosting progression from MLOps through FMOps to LLMOps.
Why it matters: The distinction identifies which operational work belongs to teams responsible for models. - Model consumer operations branch: Distinguish the model-consumer branch from model-hosting operations and explain how GenAIOps leads to ContextOps.
Why it matters: It separates operating an application built on a model from operating the model itself. - ContextOps responsibilities and ownership: Describe the responsibilities assigned here to PromptOps, RAGOps, and AgentOps, and explain why their labels and team ownership should not be treated as fixed.
Why it matters: These categories help teams assign operational responsibilities without mistaking the labels for mandatory organizational structures.
Key terms: DevOps, MLOps, FMOps, LLMOps, model hosting side, model consumer side, GenAIOps, ContextOps, context sources, PromptOps
9.3.3 LLMOps
This section explains how LLMOps adapts a foundation model for a production context and keeps its serving, caching, releases, and behavior reliable and governed.
- LLMOps scope and lifecycle: Explain when an organization needs the broader LLMOps lifecycle and why operating an LLM differs from operating a traditional predictive model.
Why it matters: The organization's degree of control determines which LLM operations it must own. - Model customization intervention layers: Distinguish the customization approaches by what each changes or learns, and describe their operational implications.
Why it matters: The intervention layer affects training cost, governance, and version management. - Serving-side cache responsibilities: Explain which cached-prefix decisions belong to LLMOps rather than the application, and why cache reuse requires isolation.
Why it matters: A useful cache must improve serving without exposing one caller's context to another. - Versioned cache warming: Describe how warming a reusable prefix after deployment or content updates affects latency and cache validity.
Why it matters: Warming avoids making the first user pay the latency of creating a shared cache entry. - Production model management controls: Describe the controls used to sustain reliability, safety, and traceability after a model reaches production.
Why it matters: A deployed model needs ongoing controls, not just a one-time release.
Key terms: LLMOps, foundation model, model serving, fine-tuning, parameter-efficient fine-tuning (PEFT), prompt learning, alignment optimization, cache-augmented generation (CAG), cached prompt prefix, time-to-live
9.3.4 PromptOps
Explains why production prompts need code-like management and describes the main PromptOps practices for changing, testing, and monitoring them.
- Prompts as behavioral configuration: Explain why PromptOps treats system and user prompt templates as configuration artifacts that need management similar to source code.
Why it matters: A prompt change can be a meaningful application change, even when no source code changes. - Production prompt change practices: Describe how versioning, automated evaluation, A/B testing, and monitoring support prompt changes in production.
Why it matters: These practices let teams detect regressions, compare alternatives, and trace the effects of deployed prompts. - Prompt regression suite coverage: Describe the behaviors and example types that a prompt regression suite should cover.
Why it matters: Varied regression cases make it more likely that a prompt change’s unwanted effects will be caught.
Key terms: PromptOps, configuration artifacts, system prompt template, user prompt template, prompt versioning, automated evaluation, golden set, A/B testing, monitoring and logging, prompt regression suites
9.3.5 RAGOps
This section explains how RAGOps keeps production RAG systems reliable through connected query-processing and data-management workflows, evaluation, and runtime monitoring.
- RAGOps purpose and workflows: Explain why RAGOps is needed in production and distinguish its two interconnected workflows: runtime query processing and data-management indexing.
Why it matters: A RAG system can lose quality through either its live query path or its underlying knowledge store. - Runtime query workflow and feedback: Trace the operations on the runtime query path and describe how telemetry feeds regression traces into automated evaluation.
Why it matters: Monitoring the live query path makes performance and answer-quality regressions available for testing. - Knowledge store update lifecycle: Describe the five data-management stages and how they check and release updates to RAG knowledge stores.
Why it matters: Knowledge-store updates need checks before release as well as a way to stay current. - Operational degradation signals: Explain why runtime monitoring includes freshness and drift signals in addition to retriever and generator metrics.
Why it matters: A stable prompt and codebase do not guarantee that a changing RAG pipeline remains healthy.
Key terms: RAGOps, runtime query-processing pipeline, data-management indexing pipeline, query reformulation, retriever execution, reranking, output validation, continuous telemetry, regression traces, ingestion and verification
9.3.6 AgentOps
This section explains why agentic systems need dedicated operations, how the AgentOps lifecycle handles failures, and how to govern tools that take external actions.
- AgentOps operational concerns: Explain why agentic systems need dedicated operations, with state management and traceability as central concerns.
Why it matters: Tracing the agent’s state and actions helps locate failures that ordinary application monitoring may miss. - Agent monitoring and checkpoints: Describe what AgentOps monitoring captures beyond traditional system signals and why it checkpoints memory and environment state.
Why it matters: An operational trace and a saved state make it possible to investigate and recover from an agent failure. - Agent failure investigation lifecycle: Trace how anomaly detection, root cause analysis, and iterative resolution follow monitoring in the four-stage AgentOps lifecycle.
Why it matters: A healthy service can still host an agent that reasons, coordinates, or acts incorrectly. - Governed external tool actions: Explain why tool use requires explicit governance and describe the controls for tools and consequential actions.
Why it matters: Tool controls limit what an agent can do outside the model and add safeguards when mistakes have real consequences.
Key terms: AgentOps, state management, traceability, monitoring, agent-specific events, checkpoints, anomaly detection, root cause analysis, resolution, tool governance
9.3.7 Context as code
This section treats context configurations as software artifacts and explains the operational practices that make their changes testable, promotable, and auditable.
- Context as code scope: Explain what it means to manage context configurations as first-class software artifacts and identify the kinds of context sources this applies to.
Why it matters: Context changes can affect an AI system even when its application code does not change. - Four operational practices: Describe the four operational practices of context as code and what each contributes.
Why it matters: These practices make context changes easier to review, reuse, validate, and release safely. - Response traceability and auditability: Explain how version-controlled context artifacts support tracing responses and reviewing changes or incidents.
Why it matters: A recorded configuration history helps teams investigate which context was in effect when a response was produced.
Key terms: context as code, context configurations, first-class software artifacts, context sources, version control, modularity and reusability, environment promotion, automated quality gates, CI/CD, version-controlling context artifacts
9.3.8 Deployment
This section explains what AI deployment includes, how deployment responsibilities vary by adoption model, and how controlled rollouts and release criteria reduce production risk.
- Scope and risk of deployment: Explain why deploying a context-engineered AI system includes more than application code and why even small context changes require care.
Why it matters: A seemingly minor context update can change production behavior, so it deserves the same release discipline as a code change. - Responsibilities by adoption model: Distinguish how deployment responsibilities expand across SaaS consumers, API integrators, LLM off-the-shelf deployments, and model hosters.
Why it matters: The adoption model determines what a team must operate and control during a release. - Controlled rollout strategies: Distinguish blue-green, canary, and shadow deployment by how each handles traffic, user exposure, and risk.
Why it matters: These strategies let teams test new behavior while controlling how much production traffic or how many users it affects. - Release metrics and rollback triggers: Describe how explicit metrics, monitoring windows, and rollback triggers govern whether a rollout continues, expands, pauses, or reverses.
Why it matters: Predefined measures and triggers make rollout decisions responsive to observed behavior rather than guesswork.
Key terms: deployment, context sources, guardrails, orchestration workflow, SaaS consumers, API integrators, LLM off-the-shelf deployments, model hosters, blue-green deployment, canary release
9.4 Hands-on
This section introduces practical examples of governance and operational controls for context-engineered systems using open-source tools.
9.4.1 Automated PII redaction with Microsoft Presidio
This section shows how Microsoft Presidio redacts sensitive information and explains what affects redaction quality.
- Presidio redaction flow: Trace how the example uses Presidio to identify and redact sensitive information in input text.
Why it matters: The two-stage flow makes clear where sensitive text is detected and where it is redacted. - Redaction quality tradeoffs: Explain what affects redaction quality and why different workflows may need different safeguards.
Why it matters: A redaction guardrail must be judged against the errors and risks of the workflow where it is used.
Key terms: Microsoft Presidio, analyzer engine, anonymizer, redaction policy, recognizer coverage, language support, false positives, false negatives, domain-specific recognizers, human review
9.4.2 Output validation with JSON Schema
Shows how structured output validation can combine schema requirements with business policy checks for AI-agent governance.
- Schema and policy validation: Explain how validating an agent’s output against a JSON Schema derived from a Pydantic model supports runtime safety checks, and describe the required output fields and policy decisions in this example.
Why it matters: Structured validation lets an application check more than whether an agent’s response contains prohibited words. - Parsing and validation retries: Describe why model output may need parsing and further attempts before validation succeeds.
Why it matters: A requested schema does not guarantee that the first model response is ready to use.
Key terms: JSON Schema, Pydantic, runtime validation, business policy checks, parsed, repair loops, retries
9.4.3 Bias detection for fairness
This section shows how group-level selection rates can reveal potential bias in credit approval recommendations and how to interpret a disparity signal cautiously.
- Group-level fairness comparison: Explain how a fairness audit compares recommendations across sensitive groups, using selection rates for young and senior credit applicants as the example.
Why it matters: An overall approval rate can hide differences between groups. - Four-fifths rule interpretation: Describe what a demographic parity ratio below 0.80 signals under the four-fifths rule and why that signal is not a final fairness judgment.
Why it matters: A threshold can flag a concern without settling whether the system is fair.
Key terms: fairness audit, sensitive attributes, MetricFrame, selection rate, four-fifths rule, demographic parity ratio
9.4.4 Basic HITL interaction pattern
This section illustrates human-in-the-loop interaction through a story-writing assistant that pauses for user input after each AI-generated paragraph.
- Story-writing approval loop: Trace how the story-writing assistant uses human input: the user supplies an initial premise, the AI generates a paragraph, and the application pauses for the user to approve it or give plot-changing instructions before the narrative proceeds.
Why it matters: The pause gives the user a chance to steer the story between AI-generated paragraphs.
Key terms: HITL, initial premise, approve, new instructions
9.4.5 AI gateway with LiteLLM
This section uses a mocked LiteLLM example to show how an AI gateway gives an application one interface while routing requests and handling provider fallback.
- Unified AI gateway interface: Explain how an AI gateway lets an application work with multiple LLM providers through one interface while centralizing routing, retries, and provider selection.
Why it matters: A single interface lets an application use multiple providers without putting provider-selection logic in the application. - Mocked provider fallback pattern: Trace the example's primary-model request and simulated failure through fallback to an alternate provider, and distinguish its mocked responses from production uses of the same architecture.
Why it matters: The example makes provider fallback concrete without requiring a live provider failure.
Key terms: AI gateway, proxy, unified interface, model-agnostic, LiteLLM, primary model, primary-path failure, fallback routing, mocked responses, alternate provider
9.4.6 Context as code with versioned prompt templates
Shows how to treat a prompt template as versioned code, test changes in CI, and turn governance policies into executable checks.
- Versioned prompt evaluation gate: Describe how the support-ticket prompt is kept in a separate, versioned Jinja2 file and checked for classification regressions. Explain that a Python evaluator renders it against test tickets with expected classifications and that GitHub Actions runs the same check in CI.
Why it matters: Versioning and testing make prompt changes reviewable and help catch classification regressions before they are accepted. - Executable governance policies: Explain the shared governance principle behind the section’s exercises: policies become checks that a pipeline executes and a reviewer can inspect, rather than remaining only in documents or being considered only at review time.
Why it matters: A policy has practical force when the system applies it as part of its operation.
Key terms: prompt_template.j2, Jinja2, source of truth, evaluation gate, GitHub Actions, CI, governance, pipeline, policy, versioned prompt templates
9.5 Summary
Where this is used later
- Governance (chapter 11) · Frames governance as a cross-cutting concern in software development.
- Context quality (chapter 12) · Governance as an explicit consideration in AI systems.
- Governance (chapter 12) · Governance concerns for AI systems.
- OpenClaw (chapter 10) · The need to consider safety and security when an AI system can access sensitive resources.
- Loop engineering (chapter 12) · The role of human review in AI-system decisions.
- Operations (chapter 11) · the general operational discipline applied here to development context
Concepts explained in this chapter
- LLMOpsOperational practices for developing, deploying, monitoring, and maintaining systems based on large language models.
- human-in-the-loopHuman oversight is the involvement of people in monitoring, reviewing, or controlling consequential AI decisions and actions.
- AI governanceGovernance comprises the structures, processes, policies, and controls that direct how AI systems are developed and operated.
- AgentOpsAgentOps is the operational discipline for governing agent tool usage, memory, state management, and orchestration.
- RAGOpsRAGOps is the operational discipline of building, deploying, and sustaining retrieval-augmented generation pipelines in production.
- promptopsPromptOps is the operational discipline of managing prompts through versioning, testing, evaluation, deployment, monitoring, and ongoing revision.
- model adaptationModel customization is the adaptation of a general-purpose model to a particular task, domain, or organizational context.
- model managementModel management is the ongoing administration of deployed models to sustain reliability, safety, and traceability.
- parameter-efficient fine-tuningParameter-efficient fine-tuning adapts a model by training a small number of parameters while keeping most base weights frozen.
- prompt injectionPrompt injection is an attack that attempts to manipulate an AI system by introducing instructions into its input or context.
- securitySecurity is the protection of systems and information from unauthorized access, disruption, or misuse.
- prompt templateA prompt template is a predefined text structure containing slots or instructions into which task-specific inputs and context are inserted.
- deployment postureAn adoption model is an organizational approach to integrating and operating generative AI within systems, processes, and governance boundaries.
- deploymentDeployment is the process of releasing an AI system or a part of it into an environment serving users or production workloads.
- privacyPrivacy is the protection of people or their information from unauthorized collection, access, use, or disclosure.
- fairnessFairness is the property of treating people and groups equitably, including avoiding unjustified disparities in system outcomes.
- context-as-codeAn approach that treats persistent instructions for AI agents as maintainable engineering artifacts.
- ContextOpsContextOps is an operational framework concerned with shaping how a model is used through its context sources.
- context assemblyContext selection is the choice of which information enters an AI system's active context.
- API integratorAn API integrator is an adoption posture in which an organization builds application-side context and orchestration pipelines that connect to hosted model endpoints.
- SaaS consumerA SaaS consumer is an adoption posture in which an organization uses vendor-hosted AI products while the provider manages the underlying stack.
- llm off-the-shelfLLM off-the-shelf is an adoption posture in which an organization deploys provider-supplied or open-weight foundation models in its managed environment.
- validationValidation checks data or operations against expected requirements before further processing.
- accountabilityAccountability is the obligation to take responsibility for AI-system decisions and their consequences.
- redactionThe removal or masking of sensitive information from data before it is shared, stored, exposed, or processed.
- monitoringMonitoring is the ongoing collection and examination of system signals and events to assess behavior and health.
- evaluation datasetAn evaluation set is a collection of test cases and expected or judged outcomes used to assess a system's performance.
- chunkingChunking is the division of large documents into smaller, coherent text segments for downstream processing and retrieval.
- indexingIndexing organizes information into a structure that supports efficient search and retrieval.
- recallRetrieval recall is the proportion of relevant items that a retrieval system successfully retrieves.
- telemetryAutomatically collected runtime evidence about a software system's operation, including events, measurements, and request-flow data.
- test suiteA test suite is a collection of software tests run to check whether a program behaves as expected.
- versioningThe practice of tracking and managing changes to files so changes can be reviewed and earlier versions restored.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
