Chapter 2: Adoption Models for GenAI
The introduction establishes why an organization must identify how it adopts GenAI before deciding how to govern it.
After this chapter you will be able to
- Adoption Path Before Governance. Explain why an organization should identify how it adopts GenAI before setting governance controls or assessing compliance needs. The adoption path affects how accountability may be assigned and how much oversight the organization must perform or demand.
- Mixed GenAI Postures. Describe GenAI posture as an organization's operational stance toward generative AI. Explain that one organization can use multiple postures at once, each with a different mix of control, complexity, and risk, and that none eliminates the need for governance.
- Three GenAI adoption postures. Distinguish SaaS Consumers, API Integrators, and Model Hosters by how the organization uses the application or model and where the model runs. Describe how questions about vendor-app use and self-hosting help classify an organization’s posture.
- Control and dependency tradeoff. Explain how organizational control and vendor dependence change across the adoption spectrum, including the added operational responsibility of hosting a model.
- Posture-specific governance priorities. Describe how the adoption posture changes the organization’s governance and compliance focus, using the section’s examples for SaaS, API integration, and hosting.
- SaaS control boundaries. Explain why SaaS is the most hands-off GenAI adoption posture and distinguish the settings an organization can configure from the application stack the vendor operates.
- Prompt controls and limits. Explain what organization-specific prompts can influence and why prompts alone cannot serve as reliable security or privacy controls.
- RAG data access boundary. Describe how SaaS RAG uses organizational data and the checks an organization should make before and during ingestion.
- Vendor validation due diligence. Explain why vendor-operated input/output validation is a trust boundary and describe what an organization should check or supplement.
- API Integrator Control Boundary. Describe what the organization operates and what it obtains from a vendor when integrating an LLM through an API.
- API Adoption Tradeoff. Explain how API integration differs from a fully vendor-controlled SaaS option in flexibility, responsibility, and governance complexity.
- Shared administration responsibilities. Explain how an API integrator administers its GenAI environment while coordinating with the vendor, and why greater customization creates governance responsibilities.
- Pre-call trust boundary controls. Trace the organization-controlled steps before an API request reaches the vendor and explain why that sequence matters.
- Vendor Request Handling. Trace how the vendor handles an outbound API request: it hosts and maintains the LLM and API endpoints, applies safety and content-moderation checks, sends the request to its own or a third-party model, and returns the output to the organization.
- Control Over Compliance Safeguards. Explain how control over the AI environment and data pipeline helps an API integrator tailor compliance and privacy safeguards.
- Responsibility Boundaries and Policy Alignment. Explain why the organization must define and validate responsibility boundaries with the vendor, and how inconsistent system-prompt policies can defeat safeguards.
- Cross-Boundary Administration. Explain why an API integrator needs centralized administration spanning internal and vendor settings, with clear roles and regular reviews of prompts and integration configurations.
- Continuous Monitoring and Log Reconciliation. Describe how cross-checking organizational call logs against vendor logs, alongside automated telemetry and alerts, supports oversight and incident detection.
- Data Hygiene and Change Tracking. Describe the explicit processes for cleaning RAG documents before ingestion, validating data boundaries and access controls, and tracking and documenting configuration changes.
- Model hoster control tradeoff. Explain what an organization controls as a model hoster and the tradeoff that control creates. Describe why hosting through a managed service can still fit this posture.
- Self-managed deployment boundary. Explain what remains within the organization’s infrastructure in a typical self-managed deployment and how that changes the organization’s control.
- Input and output pipelines. Trace a request from the organization’s custom interface through the pre-call pipeline and describe what the post-call pipeline does with the model’s response.
- Zero Trust across components. Explain why components in the self-managed architecture must verify one another and name the protections described across the flow.
- Self-hosted protection responsibility. Distinguish a self-hoster’s responsibility for protections from an API integrator’s reliance on vendor-enforced protections.
- Serving software supply chain. Explain why self-hosted serving infrastructure creates supply-chain risk and describe the precautions given here.
- Self-hosted model safeguards. Explain why a self-hoster must choose, configure, and validate model safeguards rather than assume they are built in.
- End-to-end governance ownership. Explain the tradeoff of the Model Hoster posture: greater control, privacy, and customization come with responsibility for governance and security from policy through infrastructure.
- Self-managed model safeguards. Explain why a model hoster must implement and validate its own guardrails and evaluate open models before high-risk production use.
- Shared-serving tenant isolation. Explain what a model hoster must isolate when multiple customers or business units share model serving, and where access controls must apply.
- Control and accountability shift. Explain why governance pressure points differ across SaaS, API, and full-hosting postures, and why taking on more control also means taking on more accountability.
- Goal-directed agent behavior. Distinguish an AI agent from a non-agentic chatbot by describing how an agent works toward a goal, gathers information, decides what to do next, and acts through tools or APIs with minimal human input.
- Iterative work and workflow actions. Explain how an agent can improve work over several rounds and carry a task into a business workflow, using the code-testing or report example.
- AI agent and bounded autonomy. Explain what an AI agent does, how the AI model and agent control loop contribute, and why the agent’s location and permissions limit its reach.
- Agentic workflow versus chatbot. Distinguish an agentic workflow from a chatbot that only answers questions, using the IT support example to describe adaptation across steps.
- Goal-directed planning and replanning. Explain how an agent plans steps, their order, and tool use around a goal and its constraints, then replans when a check fails or circumstances change.
- Reflection and validation checkpoints. Describe how an agent pauses to check its latest work against success criteria and decides whether to continue, revise the result, or adjust its plan.
- Observation and state updates. Explain how an agent uses scheduled or event-driven signals to update its internal state and decide what to do next.
- Actions and authorization boundaries. Distinguish committing an action from planning or observing, and explain why authorization, approval, and audit evidence matter when actions can change systems or records.
- Controlled external tool use. Trace how an agent invokes an external capability and uses its result for further reasoning or action.
- Persistent memory and data governance. Describe how memory carries an agent's state across steps or runs and why that creates retention, deletion, and access-control needs.
- Limited readiness for autonomy. Explain why organizations are cautious about granting today’s AI agents full autonomy and why human oversight remains common.
- Unnoticed autonomous error cascades. Trace how errors can cause harm between human interventions, using the Replit coding-agent incident to illustrate the risk.
- Amplified deception and misalignment. Distinguish deception that chatbots can also exhibit from the greater consequences possible when agents pursue multi-step plans with tools; explain the alignment concern illustrated by controlled tests.
- Hallucinations and weak recovery. Explain how confident false outputs, difficulty correcting mistakes, and limited context make current agents brittle over long or unexpected tasks.
- Oversight of Rapid Agent Decisions. Explain why fast, autonomous agent decisions are difficult to oversee and describe the layered controls the section suggests.
- Autonomous Harms and Organizational Accountability. Describe how an agent's ability to act can amplify privacy and bias risks, and state who remains accountable for its actions.
- Organizational Readiness for Autonomy. Explain why deploying trustworthy agents requires organizational preparation as well as technical progress.
- SaaS prototype limits. Explain why Bluebonnet could test MediAssist with synthetic patient data on a vendor-hosted chatbot but could not deploy that prototype for real clinical tasks.
- API pilot data boundary. Describe how the API pilot kept patient systems within the hospital while using an external model, and explain both its benefit and its remaining risks.
- Self-hosted control and accountability. Explain what Bluebonnet gained and took on by hosting its own model, and describe how it maintained accountability after deployment.
- Agentic monitoring workflow. Trace how the proposed MediAssist Agent would move from ongoing patient observations to an appropriate outreach or escalation.
- Agentic failure and safeguards. Trace plausible failures from observation through action, and explain why human oversight, traceability, appropriate permissions, and a mature operating environment matter.
- Agentic risk exposure. Distinguish the impact of an agentic failure from its likelihood, using MediAssist examples to explain why different deployments warrant different oversight.
Section by section
Introduction
The introduction establishes why an organization must identify how it adopts GenAI before deciding how to govern it.
- Adoption Path Before Governance: Explain why an organization should identify how it adopts GenAI before setting governance controls or assessing compliance needs. The adoption path affects how accountability may be assigned and how much oversight the organization must perform or demand.
Why it matters: Knowing the adoption path helps you ask the right governance questions before choosing controls. - Mixed GenAI Postures: Describe GenAI posture as an organization's operational stance toward generative AI. Explain that one organization can use multiple postures at once, each with a different mix of control, complexity, and risk, and that none eliminates the need for governance.
Why it matters: A single organization-wide label may miss important differences between teams' GenAI use.
Key terms: adoption path, accountability, oversight, GenAI posture, control, complexity, risk
2.1 Three postures on the adoption spectrum
The section classifies three GenAI adoption postures and shows how each shifts control, vendor dependence, and governance responsibilities.
- Three GenAI adoption postures: Distinguish SaaS Consumers, API Integrators, and Model Hosters by how the organization uses the application or model and where the model runs. Describe how questions about vendor-app use and self-hosting help classify an organization’s posture.
Why it matters: Identifying your posture is the starting point for locating the risks and responsibilities that apply to you. - Control and dependency tradeoff: Explain how organizational control and vendor dependence change across the adoption spectrum, including the added operational responsibility of hosting a model.
Why it matters: More control can reduce vendor dependence, but it also transfers work and accountability to your team. - Posture-specific governance priorities: Describe how the adoption posture changes the organization’s governance and compliance focus, using the section’s examples for SaaS, API integration, and hosting.
Why it matters: The same governance checklist will miss important differences in what your organization controls and what it entrusts to a vendor.
Key terms: SaaS Consumers, API Integrators, Model Hosters, GenAI posture, adoption spectrum, organizational control, vendor dependency, vendor due diligence, data protection agreements, data residency
2.2 SaaS or Application Consumers
This section maps the control and trust boundaries of vendor-hosted GenAI applications and explains how an organization can govern the risks it cannot control directly.
- SaaS control boundaries: Explain why SaaS is the most hands-off GenAI adoption posture and distinguish the settings an organization can configure from the application stack the vendor operates.
Why it matters: Knowing which controls are yours and which belong to the vendor is the starting point for SaaS governance. - Prompt controls and limits: Explain what organization-specific prompts can influence and why prompts alone cannot serve as reliable security or privacy controls.
Why it matters: A useful instruction to the model is not a substitute for an enforced boundary. - RAG data access boundary: Describe how SaaS RAG uses organizational data and the checks an organization should make before and during ingestion.
Why it matters: Uploading internal data creates an access boundary even when the vendor logically separates your organization’s data. - Vendor log retention trade-off: Explain the exposure created by vendor-held interaction logs and how an organization can balance reduced vendor retention with its own audit requirements.
Why it matters: Reducing a vendor’s retained data need not mean giving up records your organization must keep. - Vendor validation due diligence: Explain why vendor-operated input/output validation is a trust boundary and describe what an organization should check or supplement.
Why it matters: A safeguard operated by someone else needs scrutiny, especially when you cannot directly enforce its rules. - External model provider boundary: Trace the additional boundary between the SaaS vendor and an external LLM provider, including its implications for logs and contracts.
Why it matters: The SaaS vendor may not be the last party to process or retain an organization’s interactions.
Key terms: SaaS, trust boundaries, administrative portal, system prompts, organization-specific prompts, prompt injection, Retrieval-Augmented Generation (RAG), data ingestion, RAG layer, access controls
2.3 API Integrators
Introduces API integration as a shared-responsibility adoption model that offers more control and customization than SaaS while adding governance complexity.
- API Integrator Control Boundary: Describe what the organization operates and what it obtains from a vendor when integrating an LLM through an API.
Why it matters: The control boundary determines which parts of the system your organization operates and which remain with the model vendor. - API Adoption Tradeoff: Explain how API integration differs from a fully vendor-controlled SaaS option in flexibility, responsibility, and governance complexity.
Why it matters: Greater control brings governance responsibilities that a more vendor-controlled option may handle differently.
Key terms: API integrators, Organization Controlled Zone, Vendor Controlled Zone, LLM API, SaaS, shared responsibility, governance complexity
2.3.1 Organization-Controlled Zone: Enhanced Customization and Governance
This section explains the controls an API integrator can apply within its own environment, the responsibilities it shares with the vendor, and how vendor-hosted hybrid options shift the trust boundary.
- Shared administration responsibilities: Explain how an API integrator administers its GenAI environment while coordinating with the vendor, and why greater customization creates governance responsibilities.
Why it matters: More control does not remove the vendor from governance; it makes the division of responsibilities important. - Pre-call trust boundary controls: Trace the organization-controlled steps before an API request reaches the vendor and explain why that sequence matters.
Why it matters: The organization must apply its protections before it sends data beyond its direct control. - Post-call output governance: Describe how post-call processing and operational logging support governance after the vendor responds.
Why it matters: Governance continues after the model responds, through output handling and records of system use. - Hybrid hosting trust shift: Distinguish a vendor-hosted hybrid API option from keeping prompts and RAG data in the organization’s infrastructure, and explain the governance consequence.
Why it matters: An API endpoint does not by itself mean the organization retains control of the data and instructions behind it.
Key terms: dual-administration model, developer portal, fine-tuning, pre-call pipeline, organization-side logging, trust boundary, post-call processing, output handling standards, token consumption, hybrid model
2.3.2 Vendor-Controlled Zone: Stability and Reliability
This section traces what happens to an API request in the vendor-controlled zone, from vendor safety checks and model routing through logging and return of the output.
- Vendor Request Handling: Trace how the vendor handles an outbound API request: it hosts and maintains the LLM and API endpoints, applies safety and content-moderation checks, sends the request to its own or a third-party model, and returns the output to the organization.
Why it matters: It clarifies which parts of an API request the vendor handles and where the organization’s controls still fit. - Vendor Operational Logs: Describe why the vendor keeps operational logs and what they may contain.
Why it matters: Vendor logs are part of how requests can be monitored and incidents investigated.
Key terms: vendor-controlled zone, baseline safety policies, content-moderation filters, pre-call controls, post-call handling, operational logs, request payloads, outputs, usage metadata
2.3.3 Governance and Risk Management Opportunities
The section explains how an API integrator’s control can support compliance while creating a need to govern boundaries and agreements with the vendor.
- Control Over Compliance Safeguards: Explain how control over the AI environment and data pipeline helps an API integrator tailor compliance and privacy safeguards.
Why it matters: This control gives the organization practical ways to make its compliance safeguards fit its system. - Responsibility Boundaries and Policy Alignment: Explain why the organization must define and validate responsibility boundaries with the vendor, and how inconsistent system-prompt policies can defeat safeguards.
Why it matters: A safeguard is unreliable if it applies on one API route but is missing on another. - Vendor Service Agreements: Describe the role of SLAs and operational contracts in governing vendor-controlled processes.
Why it matters: Written service expectations help make the vendor-controlled portion governable.
Key terms: API Integrator posture, data pipeline, GDPR’s right-to-erasure, privacy safeguards, responsibility boundaries, system-prompt policies, integration points, PII redaction, prohibited-data filtering, SLAs
2.3.4 Achieving Effective Governance for an API Integrator
This section explains the administration, monitoring, and data-handling practices needed to govern an API integration across organizational and vendor boundaries.
- Cross-Boundary Administration: Explain why an API integrator needs centralized administration spanning internal and vendor settings, with clear roles and regular reviews of prompts and integration configurations.
Why it matters: Clear ownership across both sides helps prevent governance gaps in an integration. - Continuous Monitoring and Log Reconciliation: Describe how cross-checking organizational call logs against vendor logs, alongside automated telemetry and alerts, supports oversight and incident detection.
Why it matters: Joined-up visibility helps teams spot incidents and stay aware of operational and compliance risks. - Data Hygiene and Change Tracking: Describe the explicit processes for cleaning RAG documents before ingestion, validating data boundaries and access controls, and tracking and documenting configuration changes.
Why it matters: Explicit, maintained processes reduce the chance that data or configuration changes create unnoticed governance gaps.
Key terms: centralized administration, administrative roles, internal prompts, vendor integration configurations, input- and output-call logs, vendor-provided logs, monitoring, alerting, telemetry, token consumption rates
2.4 Model Hosters
This section defines the model-hoster posture as control over the inference pipeline in exchange for full security, governance, and compliance responsibility.
- Model hoster control tradeoff: Explain what an organization controls as a model hoster and the tradeoff that control creates. Describe why hosting through a managed service can still fit this posture.
Why it matters: The control a model hoster gains determines the responsibilities it must take on.
Key terms: model hosters, inference pipeline, deployment stack, guardrails, data residency, managed hosting services
2.4.1 Architecture Overview: What’s Inside the Controlled Zone
This section traces a self-managed GenAI deployment through its organization-controlled components and explains how local control and Zero Trust safeguards apply across the flow.
- Self-managed deployment boundary: Explain what remains within the organization’s infrastructure in a typical self-managed deployment and how that changes the organization’s control.
Why it matters: The location of the whole flow determines what the organization can directly oversee and safeguard. - Input and output pipelines: Trace a request from the organization’s custom interface through the pre-call pipeline and describe what the post-call pipeline does with the model’s response.
Why it matters: These pipelines show where the organization can apply checks before and after model execution. - Administrative control plane: Describe what teams manage through the administrative console and how its authority compares with an API integrator setup.
Why it matters: The control plane makes the organization’s broader operational authority concrete. - Zero Trust across components: Explain why components in the self-managed architecture must verify one another and name the protections described across the flow.
Why it matters: Keeping the deployment local does not remove the need to secure communication and access between its components.
Key terms: self-managed GenAI deployment, logging, telemetry, inference artifacts, custom-built interface, pre-call pipeline, post-call pipeline, guardrails, administrative console, control plane
2.4.2 Governance and Risk Management Opportunities
This section explains how self-hosting a model shifts responsibility for serving infrastructure, software dependencies, model safeguards, and fine-tuning risks to the organization.
- Self-hosted protection responsibility: Distinguish a self-hoster’s responsibility for protections from an API integrator’s reliance on vendor-enforced protections.
Why it matters: Who hosts the model determines who must put its protections in place. - Serving software supply chain: Explain why self-hosted serving infrastructure creates supply-chain risk and describe the precautions given here.
Why it matters: A compromised serving component can put the organization’s systems and data at risk even if the model itself works as expected. - Self-hosted model safeguards: Explain why a self-hoster must choose, configure, and validate model safeguards rather than assume they are built in.
Why it matters: Hosting a model does not guarantee that its responses will meet the organization’s safety and policy requirements. - Model provenance and fine-tuning risks: Describe the risks arising from poorly documented open-source models and from fine-tuning on poorly vetted organizational data.
Why it matters: Control over a model does not remove uncertainty about its origins or guarantee that customization preserves its safety behavior.
Key terms: API integrators, self-hosters, serving stack, core protections, serving infrastructure, supply chain, software bill of materials (SBOM), dependencies, guardrails, input validation
2.4.3 Achieving Effective Governance for a Model Hoster
Model hosting gives an organization extensive control, but makes it responsible for safeguards, model safety, serving security, tenant isolation, and governance.
- End-to-end governance ownership: Explain the tradeoff of the Model Hoster posture: greater control, privacy, and customization come with responsibility for governance and security from policy through infrastructure.
Why it matters: Hosting gives your organization more control, but also makes it accountable for the controls a vendor might otherwise provide. - Self-managed model safeguards: Explain why a model hoster must implement and validate its own guardrails and evaluate open models before high-risk production use.
Why it matters: An available guardrail or open model is not evidence that it is safe for your use case. - Fine-tuning safety checks: Describe the risks that fine-tuning on organizational data can add and the checks the team should perform before relying on the fine-tuned model.
Why it matters: Customizing a model does not preserve its previous safety behavior by default. - Serving stack supply-chain security: Describe how deployment dependencies expand a model hoster's attack surface and the practices the section recommends for managing that risk.
Why it matters: Securing the model also requires attention to the software and containers that serve it. - Shared-serving tenant isolation: Explain what a model hoster must isolate when multiple customers or business units share model serving, and where access controls must apply.
Why it matters: A shared serving system must not let one tenant retrieve another tenant's information.
Key terms: Model Hoster posture, governance, leadership, guardrails, model provenance, alignment, evaluations, fine-tuning, training data, model poisoning
2.5 Risk dimensions across deployment postures
This section maps the governance risk dimensions that change across SaaS, API, and full-hosting postures, emphasizing that greater control brings greater accountability.
- Control and accountability shift: Explain why governance pressure points differ across SaaS, API, and full-hosting postures, and why taking on more control also means taking on more accountability.
Why it matters: It helps you ask who has control and who is accountable before comparing deployment options. - Deployment-sensitive risk dimensions: Describe what the section means by risk dimensions and give examples of operational areas affected by deployment choice, such as data exposure, guardrail control, and logging.
Why it matters: These dimensions give you concrete questions to ask when comparing deployment postures.
Key terms: GenAI posture, governance pressure points, accountability, risk dimensions, likelihood × impact, data residency & exposure, guardrail control, logging & auditability
2.6 Agentic AI
This section introduces agentic AI as goal-directed systems that can take actions, improve work over multiple rounds, and connect with other tools and agents.
- Goal-directed agent behavior: Distinguish an AI agent from a non-agentic chatbot by describing how an agent works toward a goal, gathers information, decides what to do next, and acts through tools or APIs with minimal human input.
Why it matters: This distinction explains why agents can affect systems beyond the conversation itself. - Iterative work and workflow actions: Explain how an agent can improve work over several rounds and carry a task into a business workflow, using the code-testing or report example.
Why it matters: The examples show why agentic use can change both the quality of a result and what happens after it is produced. - Agent interoperability efforts: Describe the different connections addressed by MCP, A2A, and Agent Skills, while noting that agent adoption remains at an early stage.
Why it matters: These distinctions show what kinds of connections the emerging agent ecosystem is trying to make easier.
Key terms: agentic AI, AI agent, non-agentic generative AI, external tools and APIs, agentic way, business workflows, Model Context Protocol (MCP), Agent-to-Agent (A2A) protocol, Agent Skills
2.6.1 What Does “Agentic” Mean?
Defines AI agents, agentic workflows, and multi-agent systems, showing how autonomy, permissions, and collaboration shape what they can do and where risks arise.
- AI agent and bounded autonomy: Explain what an AI agent does, how the AI model and agent control loop contribute, and why the agent’s location and permissions limit its reach.
Why it matters: Where an agent runs and what it can access determine both the actions it can take and the risks it creates. - Agentic workflow versus chatbot: Distinguish an agentic workflow from a chatbot that only answers questions, using the IT support example to describe adaptation across steps.
Why it matters: The distinction clarifies when a system is taking and adjusting actions rather than simply answering a request. - Multi-agent collaboration and tradeoff: Describe how a multi-agent system divides work among agents and the benefit and drawback of their collaboration.
Why it matters: Collaboration can improve a result, but coordinating agents introduces additional failure opportunities.
Key terms: AI Agent, control loop, bounded autonomy, permissions, Agentic Workflow, IT Support agent, Multi-Agent System, orchestrating multiple agents
2.6.2 Common properties of Agentic AI
This section explains how agents pursue goals across multiple steps through planning, reflection, observation, action, tool use, and persistent memory, and why those properties create governance needs.
- Goal-directed planning and replanning: Explain how an agent plans steps, their order, and tool use around a goal and its constraints, then replans when a check fails or circumstances change.
Why it matters: A plan makes the agent's route, limits, and possible changes visible for review. - Reflection and validation checkpoints: Describe how an agent pauses to check its latest work against success criteria and decides whether to continue, revise the result, or adjust its plan.
Why it matters: Reflection gives an agent a chance to catch a bad result before it carries that result forward. - Observation and state updates: Explain how an agent uses scheduled or event-driven signals to update its internal state and decide what to do next.
Why it matters: An agent that observes changing conditions need not keep acting on stale information. - Actions and authorization boundaries: Distinguish committing an action from planning or observing, and explain why authorization, approval, and audit evidence matter when actions can change systems or records.
Why it matters: The point where an agent commits a change is where mistakes may become hard to undo. - Controlled external tool use: Trace how an agent invokes an external capability and uses its result for further reasoning or action.
Why it matters: Tool calls connect an agent's reasoning to live data and external systems, so permission checks are consequential. - Persistent memory and data governance: Describe how memory carries an agent's state across steps or runs and why that creates retention, deletion, and access-control needs.
Why it matters: Remembering across runs helps an agent continue work but also extends how long and where data may persist.
Key terms: single-shot, planning, constraints, success criteria, stop conditions, re-planning, reflection, self-refining, checkpoints, validator tools
2.6.3 Why Agentic AI is still experimental
This section explains why current AI agents remain experimental: autonomy can amplify errors and misaligned behavior, while proposed reliability improvements still need validation.
- Limited readiness for autonomy: Explain why organizations are cautious about granting today’s AI agents full autonomy and why human oversight remains common.
Why it matters: This is the section’s reason for treating greater agent autonomy as a cautious decision rather than a default. - Unnoticed autonomous error cascades: Trace how errors can cause harm between human interventions, using the Replit coding-agent incident to illustrate the risk.
Why it matters: The Replit incident makes the consequences of unsupervised agent actions concrete. - Amplified deception and misalignment: Distinguish deception that chatbots can also exhibit from the greater consequences possible when agents pursue multi-step plans with tools; explain the alignment concern illustrated by controlled tests.
Why it matters: This explains why misleading behavior warrants particular concern when a system can continue taking actions on its own. - Hallucinations and weak recovery: Explain how confident false outputs, difficulty correcting mistakes, and limited context make current agents brittle over long or unexpected tasks.
Why it matters: These technical limitations explain why plausible-looking agent work cannot yet be assumed to remain reliable through a task. - Experimental self-correction research: Describe the promise of reflection and self-critique approaches while stating their current production limitation.
Why it matters: Promising laboratory results should not be mistaken for a solved production reliability problem.
Key terms: fully autonomous AI agents, human oversight, Levels 1–2, reliability, human interventions, guardrails, deception, manipulation, alignment, in-context scheming
2.6.4 Governance Challenges
This section explains why autonomous AI agents challenge organizational oversight, explainability, accountability, and readiness for deployment.
- Oversight of Rapid Agent Decisions: Explain why fast, autonomous agent decisions are difficult to oversee and describe the layered controls the section suggests.
Why it matters: Controls must give an organization meaningful oversight even when people cannot review every decision as it happens. - Limits of Visible Reasoning: Explain why an agent showing its reasoning does not necessarily satisfy explainability expectations.
Why it matters: A plausible-looking explanation is not enough to establish how an agent reached a consequential decision. - Autonomous Harms and Organizational Accountability: Describe how an agent's ability to act can amplify privacy and bias risks, and state who remains accountable for its actions.
Why it matters: Delegating an action to an agent does not delegate away the organization's responsibility for its effects. - Organizational Readiness for Autonomy: Explain why deploying trustworthy agents requires organizational preparation as well as technical progress.
Why it matters: An agent's technical capability does not by itself make an organization ready to deploy it safely.
Key terms: oversight, multi-layered controls, human approval, audit, domain constraints, chain-of-thought prompting, transparency, explainability, data privacy, algorithmic bias
2.7 MediAssist: A case study in AI governance
MediAssist illustrates how a healthcare chatbot’s governance needs change as it moves from a SaaS prototype to an API pilot, self-hosted production, and a possible agentic extension.
- SaaS prototype limits: Explain why Bluebonnet could test MediAssist with synthetic patient data on a vendor-hosted chatbot but could not deploy that prototype for real clinical tasks.
Why it matters: A successful proof of concept is not necessarily suitable for real patients. - API pilot data boundary: Describe how the API pilot kept patient systems within the hospital while using an external model, and explain both its benefit and its remaining risks.
Why it matters: Moving the interface and records in-house changes what the vendor receives, but does not eliminate the risks of an external API. - Self-hosted control and accountability: Explain what Bluebonnet gained and took on by hosting its own model, and describe how it maintained accountability after deployment.
Why it matters: Fuller control also means owning the work of keeping a clinical system safe and auditable. - Agentic monitoring workflow: Trace how the proposed MediAssist Agent would move from ongoing patient observations to an appropriate outreach or escalation.
Why it matters: Continuous monitoring and the ability to initiate action make this extension different from a chatbot awaiting a patient’s question. - Agentic failure and safeguards: Trace plausible failures from observation through action, and explain why human oversight, traceability, appropriate permissions, and a mature operating environment matter.
Why it matters: An agent can carry an early mistake into a consequential clinical action. - Agentic risk exposure: Distinguish the impact of an agentic failure from its likelihood, using MediAssist examples to explain why different deployments warrant different oversight.
Why it matters: Oversight should reflect what an agent can affect and how likely it is to fail, not just whether it is called agentic.
Key terms: SaaS prototype, synthetic medical data, HIPAA Business Associate, API pilot, de-identified data, zero-retention, re-identification, self-hosted production, model card, system factsheet
2.8 Summary
The section summarizes how deployment posture changes an organization’s control over GenAI risks, while agentic capability adds a separate source of risk that governance controls must address.
Where this is used later
- Untrustworthy Input (chapter 4) · Provides the reason untrustworthy input can undermine prompt-based instructions.
- Data Security (chapter 4) · Builds on the need to protect data made available to an AI system.
- Storage and Memorization (chapter 5) · Connects vendor retention choices to privacy risks around stored data.
- Vendor security gaps (chapter 4) · Provides the SaaS example of dependence on vendor security controls.
- Vendor’s Inappropriate Use of Organizational Data (chapter 5) · Provides context for examining vendor use and sharing of organizational data.
- Adoption Model Differences (chapter 5) · Revisits how adoption models affect privacy responsibilities.
- Operations & Monitoring (Level 5) (chapter 3) · Extends the need for monitoring into governance during operations.
- Insecure Environments (chapter 4) · Provides the baseline inter-component security assumptions for examining insecure environments.
Concepts explained in this chapter
- human-in-the-loopAn approach in which a person participates in an AI system’s operation, such as by reviewing or authorizing its outputs or actions.
- memoryAgent memory is information retained by an agent for use in later interactions or tasks.
- SaaS consumerA SaaS consumer is an organization that uses generative AI through a vendor-provided application or platform.
- AI agentAn AI agent is a software system that can plan and perform tasks, including using tools and taking actions on a user's behalf.
- reliabilityReliability is the degree to which a system performs consistently and correctly under expected conditions.
- model hoster postureA model hoster is an organization that runs a generative AI model on infrastructure it controls or manages.
- API integratorAn API integrator is an organization that incorporates a third-party generative AI model into its systems through an API.
- planningPlanning is the process by which an agent selects and orders actions, tools, constraints, success criteria, and stop conditions to pursue a goal.
- trust boundaryA trust boundary is a division between components or parties across which control, responsibility, or security assumptions change.
- validation layerInput/output validation is the inspection of AI requests and responses to sanitize them or detect content that violates defined requirements.
- logLogging is the recording of system events and associated information for monitoring, audit, or analysis.
- multi-agent systemA multi-agent system is a system in which multiple agents interact or collaborate to perform tasks.
- self-hosted productionA self-managed GenAI deployment is an arrangement in which an organization hosts and operates the generative-AI system and its supporting processes.
- monitoringObservation of a system or process to track its status, behavior, or performance.
- biasA systematic distortion in data or model behavior that can produce unfair or skewed outcomes.
- control loopA control loop is a process in which a system uses feedback from its actions or environment to guide subsequent behavior.
- autonomy-level classificationA grade describing how independently an AI system or process can operate from human direction.
- deceptionDeception is behavior that intentionally or strategically misleads another party about facts, actions, or intentions.
- approval gateAn approval gate is a control that requires authorization before an action proceeds.
- audit trailAn audit log is a chronological record of events and actions that supports review, accountability, or compliance.
- deletionA procedure for removing stored data and tracing its removal across relevant systems or records.
- working memoryTemporary agent state holding information for the current task, such as sub-goals, partial outputs, and tool results.
- anonymizationAnonymization is the alteration of information to prevent it from being linked to an identifiable person.
- data retentionData retention is the storage of data for a defined period under operational, legal, or policy requirements.
- sanitizationInput sanitization is the modification of supplied data to remove or neutralize content that could violate security or processing requirements.
- content filteringAn output filter is a control that checks or restricts generated content before delivery.
- interpretabilityThe ability to provide an understandable account of how an AI system reaches its decisions or outputs.
- transparencyTransparency is the availability of understandable information about an AI system’s operation and decision-making.
- authorizationAuthorization determines which actions or resources an entity is permitted to access.
- guardrailA control that constrains AI system inputs, outputs, or interactions to reduce safety, security, or policy violations.
- rate limitA rate limit is a configured bound on the number or frequency of requests permitted during a specified period.
- secret managementSecret management is the secure storage, controlled use, and periodic replacement of credentials, keys, and other sensitive authentication material.
- token limitA token cap is a configured maximum on the number of language-model tokens that may be processed or generated for an interaction or session.
- MCPMCP is a protocol for connecting AI applications to external tools and data sources.
- fine-tuningFine-tuning is the additional training of a pretrained model on task-specific or organization-specific data to adapt its behavior or capabilities.
- tenant isolationTenant isolation is the separation of data, state, and operations belonging to different customers or organizational tenants in shared infrastructure.
- alertingThe process of notifying designated recipients when specified events, conditions, or thresholds occur.
- auditabilityAuditability is the degree to which actions, decisions, and system behavior can be examined and verified from available records and evidence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
