Chapter 4: Securing GenAI
The chapter introduction explains why LLMs are vulnerable to instructions hidden in data, why filters cannot fully enforce the instruction–data boundary, and why greater AI capabilities raise the security stakes.
After this chapter you will be able to
- Instruction and data boundary. Explain why an LLM may treat attacker-controlled content as instructions, unlike traditional software that can enforce a structural boundary between commands and data.
- Capability and security stakes. Describe how moving from answering questions to accessing data and taking actions makes an AI system’s mistakes or manipulation more consequential.
- Compounding Risk in ClaimAssist. Trace ClaimAssist’s evolution from a policy FAQ chatbot to a customer-data-connected assistant that can take actions, with an agentic version under consideration. Explain that each added capability brings value while opening new attack paths and failure modes.
- Environment and model exposure. Explain how weak surrounding security and an unvetted or changed model can put a GenAI application at risk even when another part of the system appears secure.
- Untrusted input and data access. Describe how untrusted content creates opportunities for prompt injection and how runtime access to sensitive data raises the potential impact.
- Consequential actions and agency. Distinguish an application that can make real-world changes from an agent that also chooses its own steps toward a goal, and explain the added exposure of each.
- Dependency ladder and severity. Apply the six-factor ladder to identify a system’s exposures, and explain why its position is not itself a severity score.
- Cumulative ClaimAssist Risk Ladder. Explain which risks apply regardless of ClaimAssist’s capabilities, how new capabilities add risks, and why controls must match the resulting exposure.
- Action-Enabled State Changes. Explain why Version 5’s ability to change records raises the stakes beyond incorrect answers or data leakage.
- Autonomous Planning Data Exposure. Explain how Version 6’s self-directed planning can expose another customer’s data even while pursuing a legitimate request.
- Cross-Agent Trust Vulnerability. Trace how hidden instructions in a claim description can pass from ClaimAssist to another agent and expose unrelated claimants’ information.
- GenAI Security Blast Radius. Explain why conventional security controls remain necessary for GenAI systems and how a leaked model key can have a broader impact than a leaked database credential.
- Incident traceability requirements. Explain why insufficient logging prevents investigators from reconstructing a GenAI incident or determining who was affected, and why GenAI needs records beyond an application's input and output.
- Baseline observable event trail. Describe the baseline events to log for a GenAI system handling sensitive information and how a shared request identifier connects them from prompt to downstream action.
- Privacy protections for logs. Explain the privacy risk of comprehensive audit logs and the protections the section prescribes before storing sensitive content or retaining raw data.
- Vendor infrastructure and insider exposures. Explain how shared infrastructure can expose customer data and how unauthorized insider changes can alter a vendor model’s responses, using the incidents in the section.
- Testable vendor security assurances. Describe the specific questions an organization should ask when relying on a vendor model rather than accepting a claim that the vendor does not store data.
- Three Supply-Chain Risk Layers. Explain why an attacker can compromise a GenAI system without attacking the model, and name the three layers of surrounding software that create this risk.
- Trusted Extensions and Services. Explain how permissions and trust granted to plugins, MCP integrations, and model-file services can turn their failures or malicious updates into security breaches.
- Library Risks Beyond CVEs. Distinguish the risks found by conventional vulnerability scanners from other AI-library and model-file risks described here.
- Supply-Chain Governance Controls. Describe the section’s baseline controls and how its advanced measures extend visibility and assurance over third-party components.
- Denial of wallet risk. Explain why overwhelming a GenAI endpoint threatens both availability and finances, and how runaway loops or targeted attacks can multiply token usage.
- Baseline consumption limits. Describe the baseline controls for limiting requests, tokens, query demands, spending, and multi-agent loops. Distinguish a hard spend limit from a budget alert.
- Impact of Leaked AI Credentials. Explain why a leaked GenAI API key can expose more than one application and how attackers can exploit it.
- Expanding Key Permissions. Explain how an existing key can become more sensitive when a platform adds a service, and how scoped keys limit that risk.
- Baseline Secret Management. Describe the baseline practices for keeping AI credentials out of content, limiting their lifetime, and managing their use centrally.
- Shadow AI and data exposure. Explain what shadow AI is, why employees use it, and how personal or unmanaged tools can expose organizational data.
- Unapproved production call paths. Describe how shadow AI can appear in production code and why a direct API call can defeat controls even when its outputs look the same.
- Baseline and advanced controls. Distinguish the section’s baseline responses to shadow AI from its advanced controls for accounts, data, and LLM traffic.
- Limits of Execution Isolation. Explain why isolating a model can protect its host server without protecting the application from the model’s own logic.
- Risks Embedded in Model Weights. Explain why inspecting a model file cannot reveal every vulnerability, and distinguish weight-based risks from malicious code that executes when a file is loaded.
- Fine-Tuning Safety Degradation. Explain how fine-tuning for a domain can weaken a model’s existing safety behaviors, even with ordinary or proprietary training data, and why the resulting model still needs testing.
- Trigger-Activated Model Backdoors. Explain how poisoning places a backdoor in model weights and why the model can appear normal until a specific trigger activates harmful behavior.
- Limits of Safety Training. Describe what the sleeper-agent research showed about detecting or removing backdoors with standard safety interventions, while distinguishing that demonstration from a production attack.
- Poisoned Model Supply Chain. Explain why organizations obtaining models or fine-tuned weights from others face a poisoning risk, using the low-cost attack finding and the typosquatted model demonstration.
- Model artifact theft. Explain why model theft matters even to organizations that only fine-tune models, and describe what happens when an attacker gains access to stored model weights.
- Functional model extraction. Describe how repeated queries to a private model’s API can be used to train a student model, and distinguish this from stealing the model weights.
- External model file checks. Describe how to vet an online model before use by checking its publisher, repository, accompanying files, release hash, and model-file format, while recording its provenance and approval.
- Repeatable behavioral model tests. Explain why each new or updated model should undergo automated, repeatable behavioral checks in addition to static scanning.
- Prompt injection and jailbreaks. Explain how prompt injection makes a model follow an attacker’s instructions instead of its intended purpose, and distinguish a jailbreak as a variant aimed at the model’s safety guardrails.
- Limits of input guardrails. Describe what input guardrails screen and why they cannot be the only defense against prompt injection.
- Consequences of combined risk factors. Explain why the same prompt injection can have more serious consequences when a system has data access or the ability to act.
- Indirect prompt injection paths. Explain how instructions hidden in material an AI assistant is asked to process can redirect it, even when the user did not type the instructions into chat. Describe how the GitLab Duo or Amazon Q example led to disclosure of information the assistant could access.
- Persistent rules file backdoors. Distinguish a poisoned AI rules file from a one-off indirect injection, including why hidden characters make the instructions easy to overlook and why the effects can spread.
- AI configuration file safeguards. Describe the baseline review and hidden-character checks for AI configuration files, and explain why agent instruction files warrant security-aware approval, particularly in repositories accepting outside contributions.
- Data Leak Lethal Trifecta. Explain why prompt injection alone does not cause a data leak, and describe the three conditions that make one possible. Explain why breaking any one condition stops the attack, while useful assistants often need all three.
- Confused Deputy Data Exposure. Explain how a GenAI assistant using a high-privilege service account can return enterprise information that the requesting user could not access directly.
- User Context Permission Checks. Describe user-context queries as the secure alternative and explain what must carry the requesting user's identity.
- Latent Oversharing Exposure. Explain why an AI system can expose sensitive documents even when it correctly follows existing access controls. Describe how meaning-based retrieval makes overshared files easier to discover than keyword search did.
- Stale Permission Checks. Explain how indexed access rights can become stale and how real-time external authorization addresses that gap, including its latency and permission-boundary limitations.
- Layered Retrieval Boundaries. Distinguish tenant isolation from metadata filtering and describe how production retrieval systems combine them with real-time checks. State the factors that shape the balance.
- Retrieved Content as Instructions. Trace how an attacker places hidden instructions in searchable content and how a later user query can cause the assistant to treat that content as instructions rather than reference material.
- Poisoning Defenses and Limits. Describe how deployments can reduce the retrieval and influence of poisoned content, and explain why those measures do not guarantee safety.
- Write Access and Unauthorized Actions. Explain how write access worsens the confused deputy problem: an assistant using a shared, high-privilege identity may perform actions the requesting user could not authorize, with potentially irreversible effects.
- Wrong Target Despite Correct Permissions. Describe how an AI can carry out an allowed action against the wrong record when it misinterprets an ambiguous request or input, and explain why submitting such mistakes at scale matters.
- Operation Scoped Write Credentials. Explain why AI should receive short-lived write credentials limited to the current action and target, rather than standing write access. Describe how that limit contains actions outside the authorized scope even if prompt injection redirects the system.
- Preexecution Checks and Tiered Approval. Describe how execution requirements are validated before an action and how user confirmation becomes more demanding as the action’s impact rises.
- Excluded Actions and Human Oversight. Distinguish actions that should be unavailable to AI regardless of user requests from actions kept under human oversight. Explain the difference between human-in-the-loop and human-on-the-loop.
- Full Intent Chain Audit. Describe what must be recorded for every state-changing operation and why logging only the resulting update is insufficient.
- Scripted versus agentic action. Distinguish ClaimAssist Version 5’s user-requested, predefined workflow from Version 6’s ability to decide its own steps for a complex claim.
- Planning-related security risks. Explain why autonomy creates security risks beyond those addressed by controls for predefined actions, and name the kinds of manipulation the section identifies.
- Tool Misuse Within Permissions. Explain how an attacker can manipulate an agent into misusing tools it is already permitted to use, and why permission checks alone may miss the abuse.
- Persistent Memory Poisoning. Describe how malicious instructions placed in an agent’s memory can affect later interactions and why that persistence complicates auditing.
- Cascading Multi-Agent Failures. Trace how an error or manipulation in one agent’s output can spread through other agents, including when no attacker is involved.
- Rogue Agent Actions. Explain how an agent acting on a poorly specified objective can cause damage even without a deliberate attack.
- Browser Agent Lethal Trifecta. Explain how browser agents create the Lethal Trifecta while working across authenticated web sessions.
- Browser Agent Risk Controls. Describe how an organization that permits browser agents can reduce their access to sensitive data and observe their actions.
- MCP Integration and Shared Risk. Explain how MCP servers let compatible agents use external tools without bespoke connectors, and why a vulnerable server or malicious content in one connected system can put other connected systems at risk.
- Tool Permissions and Poisoning. Describe how MCP tools can request more access than they need and how malicious instructions hidden in a tool description can influence a naive agent.
- Dynamic Tool Discovery. Explain why approving an MCP server's initial tools does not necessarily approve everything it may later offer, and describe the safeguards given in the section.
- Authentication and Access Accountability. Explain why optional authentication and widespread use of unauthenticated servers or long-lived static secrets make it difficult to tell who accessed data through an MCP-connected agent.
- Tool Sprawl and Oversight. Explain why unvetted MCP integrations create a security problem and how an approved tool registry gives IT central oversight.
- Overpermissioned Server Exposure. Explain why even official MCP servers may give a compromised agent access to more operations than it needs.
- Per Tool Registry Limits. Describe the limits and responsibilities an effective registry records for each approved tool.
- Gateway Enforcement Layer. Explain why a registry needs an enforcement layer and how a gateway applies controls to agent-to-tool requests.
- External tool-call decision path. Trace how an external policy engine decides whether an agent’s proposed tool call proceeds.
- Calibrated human approval thresholds. Explain why agentic systems need approval thresholds based on categories of action rather than confirmation of every individual tool call. Describe how too many low-priority requests can make human review ineffective.
- Read-like outbound exfiltration. Explain how an apparently read-only agent operation can leak sensitive data, and distinguish ordinary reads from outbound calls that need monitoring and restriction.
- Defense in depth rationale. Explain why ClaimAssist uses layered controls even with policy enforcement and human oversight.
- Sandbox boundaries and limits. Describe how sandboxing confines ClaimAssist’s agent and why its boundaries must be chosen carefully.
- Memory isolation and persistence. Explain how ClaimAssist limits persistent-memory attacks while retaining useful context, and how summarizing memory writes adds protection.
- Agent behavioral monitoring. Describe the agent-specific behavior that should prompt investigation and the value of capturing reasoning traces.
- Graduated response and kill switch. Distinguish graceful degradation from an emergency kill switch and explain why the latter needs automated and manual triggers.
- Agent identity and credentials. Explain how to make each agent and its credentials accountable, and distinguish short-lived credentials from advanced just-in-time access.
- External tool authorization and approval. Explain why tool authorization belongs outside the agent’s reasoning and how tool consequences determine human approval. Describe the additional control an MCP gateway provides.
- Isolation and layered protection. Describe how separating permissions and restricting execution reduce exposure to untrusted content, including the special precautions for browser agents.
- Agent visibility and containment. Explain what to record to investigate agent behavior, how advanced anomaly detection uses that visibility, and how a kill switch stops a compromised agent.
- Customer Data Access Boundaries. Explain how ClaimAssist limited access when connected to customer records and how the team checked the safety effects of fine-tuning.
- Identity Bound Write Actions. Explain how ClaimAssist controlled actions once it could change claim records and trigger communications.
- Agentic Tool And Memory Controls. Explain why autonomous planning required controls outside ClaimAssist’s reasoning and how its memory was contained.
- Governed Autonomy And Shadow AI. Describe how ElectraMotors responded to unsanctioned employee AI use and set a limited initial scope for agentic autonomy.
What it builds on
- What Does “Agentic” Mean? (chapter 2) · The earlier introduction to what it means for AI to be agentic.
- Common properties of Agentic AI (chapter 2) · Introduces properties of agentic AI.
- Key artifacts (chapter 3) · The operational artifacts used to trace a GenAI system.
- Three postures on the adoption spectrum (chapter 2) · Uses the distinction between GenAI adoption postures.
Section by section
Introduction
The chapter introduction explains why LLMs are vulnerable to instructions hidden in data, why filters cannot fully enforce the instruction–data boundary, and why greater AI capabilities raise the security stakes.
- Instruction and data boundary: Explain why an LLM may treat attacker-controlled content as instructions, unlike traditional software that can enforce a structural boundary between commands and data.
Why it matters: This distinction explains why malicious instructions in something as ordinary as an email can affect an AI assistant. - Limits of security guardrails: Distinguish useful prompt filters and traditional security controls from an enforceable boundary that determines whether an instruction is legitimate.
Why it matters: It helps you value existing controls without mistaking them for a complete solution to prompt injection. - Capability and security stakes: Describe how moving from answering questions to accessing data and taking actions makes an AI system’s mistakes or manipulation more consequential.
Why it matters: The consequences of an attack depend on what the AI system is able to access and do.
Key terms: LLMs, instructions, data, prompt injection, guardrails, input validation, traditional security controls, data access, action capabilities, agentic behavior
4.1 ClaimAssist
Introduces ClaimAssist’s growing capabilities as a case study in how GenAI security risk compounds.
- Compounding Risk in ClaimAssist: Trace ClaimAssist’s evolution from a policy FAQ chatbot to a customer-data-connected assistant that can take actions, with an agentic version under consideration. Explain that each added capability brings value while opening new attack paths and failure modes.
Why it matters: ClaimAssist shows why security risk must be reconsidered as an assistant gains capabilities.
Key terms: ClaimAssist, FAQ chatbot, agentic version, attack paths, failure modes
4.2 Risk Factors
The section introduces six exposure factors and a dependency ladder for identifying a GenAI system’s security risks without mistaking the ladder for a severity ranking.
- Environment and model exposure: Explain how weak surrounding security and an unvetted or changed model can put a GenAI application at risk even when another part of the system appears secure.
Why it matters: Securing the model alone does not secure the application it runs in. - Untrusted input and data access: Describe how untrusted content creates opportunities for prompt injection and how runtime access to sensitive data raises the potential impact.
Why it matters: The same malicious input can have much greater consequences when the application can retrieve sensitive information. - Consequential actions and agency: Distinguish an application that can make real-world changes from an agent that also chooses its own steps toward a goal, and explain the added exposure of each.
Why it matters: Acting and autonomously deciding how to act create different exposures beyond producing text. - Dependency ladder and severity: Apply the six-factor ladder to identify a system’s exposures, and explain why its position is not itself a severity score.
Why it matters: The ladder helps locate exposure without hiding a serious low-level weakness behind a system’s apparent simplicity.
Key terms: Environment, Model, fine-tuning, vendor security, Input, Data access, prompt injection, RAG pipelines, Ability to make changes, Agency
4.2.1 ClaimAssist and How Risk Factors Stack
ClaimAssist’s successive capabilities show how security risks accumulate and why controls must grow with the system’s exposure.
- Cumulative ClaimAssist Risk Ladder: Explain which risks apply regardless of ClaimAssist’s capabilities, how new capabilities add risks, and why controls must match the resulting exposure.
Why it matters: A control set suited to an earlier version of ClaimAssist may no longer match what a later version can expose or do. - Personalized Assistant Profile Extraction: Describe how data access in Version 3 lets an attacker with a compromised customer session extract a consolidated profile.
Why it matters: Adding data access can make an existing account takeover much more effective. - Fine-Tuned Model Concerns: Describe the additional model-level concerns in Version 4 while recognizing that it retains Version 3’s data risks.
Why it matters: Customizing a model does not remove existing data risks and may introduce new ways for sensitive material to surface. - Action-Enabled State Changes: Explain why Version 5’s ability to change records raises the stakes beyond incorrect answers or data leakage.
Why it matters: When ClaimAssist can act, an attacker can turn compromised access into changes with direct financial consequences. - Autonomous Planning Data Exposure: Explain how Version 6’s self-directed planning can expose another customer’s data even while pursuing a legitimate request.
Why it matters: A helpful plan can cross data boundaries when the agent’s access is broader than the customer’s request warrants. - Cross-Agent Trust Vulnerability: Trace how hidden instructions in a claim description can pass from ClaimAssist to another agent and expose unrelated claimants’ information.
Why it matters: A request passed between agents can carry attacker-controlled instructions into a second agent’s access.
Key terms: risk ladder, environment risk, model risk, input risk, controls, exposure, Version 3, Personalized Assistant, account takeover, consolidated dossier
4.3 Insecure Environments
This section explains why securing the underlying environment matters before addressing GenAI-specific threats: familiar security gaps can have a wider impact in a GenAI system.
- GenAI Security Blast Radius: Explain why conventional security controls remain necessary for GenAI systems and how a leaked model key can have a broader impact than a leaked database credential.
Why it matters: Securing the environment limits the reach of a failure before you tackle GenAI-specific threats.
Key terms: underlying environment, blast radius, model key, classic security
4.3.1 Insufficient Logging
This subsection explains how GenAI audit logging supports incident investigation, what a baseline should capture, and how to limit the privacy risks that logging creates.
- Incident traceability requirements: Explain why insufficient logging prevents investigators from reconstructing a GenAI incident or determining who was affected, and why GenAI needs records beyond an application's input and output.
Why it matters: A usable incident record lets a team investigate harm or downstream failure rather than guess at the cause and impact. - Baseline observable event trail: Describe the baseline events to log for a GenAI system handling sensitive information and how a shared request identifier connects them from prompt to downstream action.
Why it matters: Connected event records make it possible to follow one request through the system during an investigation. - Privacy protections for logs: Explain the privacy risk of comprehensive audit logs and the protections the section prescribes before storing sensitive content or retaining raw data.
Why it matters: Incident visibility should not turn audit logs into an unnecessary store of sensitive information. - Advanced decision records: Distinguish advanced Decision Records from a record of actions by describing the context they capture for a model response.
Why it matters: A record of the basis for a response helps answer auditors' and regulators' questions when something goes wrong.
Key terms: audit logging, traceability, reasoning chain, observability, shared request identifier, token counts, PII redaction, tokenization, retention, Decision Records
4.3.2 Vendor security gaps
The section shows how vendor infrastructure and insider access can expose data or alter model behavior, and why vendor security assurances need specific scrutiny.
- Vendor infrastructure and insider exposures: Explain how shared infrastructure can expose customer data and how unauthorized insider changes can alter a vendor model’s responses, using the incidents in the section.
Why it matters: These examples show that vendor risk includes both data exposure and changes to what a model says. - Testable vendor security assurances: Describe the specific questions an organization should ask when relying on a vendor model rather than accepting a claim that the vendor does not store data.
Why it matters: Specific questions make vendor security claims possible to evaluate.
Key terms: tenant isolation, shared in-memory infrastructure, insider access, system prompt, caches, system instructions, audited, disclose and remediate incidents
4.3.3 Supply-Chain Risks
This section explains how trusted software around a GenAI model creates supply-chain risks across three layers, and how to govern those risks.
- Three Supply-Chain Risk Layers: Explain why an attacker can compromise a GenAI system without attacking the model, and name the three layers of surrounding software that create this risk.
Why it matters: Securing the model alone leaves trusted routes into the system unexamined. - Trusted Extensions and Services: Explain how permissions and trust granted to plugins, MCP integrations, and model-file services can turn their failures or malicious updates into security breaches.
Why it matters: A functional-looking integration can still expose data through a bug or later update. - Library Risks Beyond CVEs: Distinguish the risks found by conventional vulnerability scanners from other AI-library and model-file risks described here.
Why it matters: A clean known-vulnerability scan does not establish that an AI dependency or model file is safe. - Infrastructure Exposure and Ownership: Describe how compromised foundation infrastructure can expose GenAI data or alter a system, and distinguish who manages that risk in SaaS versus a self-run API integration.
Why it matters: Deployment choices determine whether you must operate the infrastructure controls or rely on a vendor to do so. - Supply-Chain Governance Controls: Describe the section’s baseline controls and how its advanced measures extend visibility and assurance over third-party components.
Why it matters: These controls make third-party components visible and reduce the chance that a compromised dependency changes production unnoticed.
Key terms: GenAI supply chain, extensions and tools, AI-specific libraries, foundation infrastructure, plugins, Model Context Protocol (MCP), platform service, update process, CVE, transitive libraries
4.3.4 Uncontrolled usage and denial of service
This section explains how uncontrolled GenAI resource use can disrupt service and drive up costs, and describes controls for limiting and detecting it.
- Denial of wallet risk: Explain why overwhelming a GenAI endpoint threatens both availability and finances, and how runaway loops or targeted attacks can multiply token usage.
Why it matters: A service can keep consuming expensive resources even when it is no longer doing useful work. - Baseline consumption limits: Describe the baseline controls for limiting requests, tokens, query demands, spending, and multi-agent loops. Distinguish a hard spend limit from a budget alert.
Why it matters: Limits that actually stop consumption protect a project in ways that alerts alone cannot. - Advanced cost safeguards: Describe how real-time monitoring, automated circuit breakers, capacity planning, and denial-of-wallet simulations strengthen consumption controls.
Why it matters: Testing and automatic interruption can expose or stop costly failures before they escalate in production.
Key terms: availability attacks, denial of wallet, runaway agent loop, exhaustion attacks, rate-limit, token quotas, timeouts, hard spend limits, budget alerts, supervisor
4.3.5 Improper secret management
This section explains how GenAI credentials leak, why their exposure can have a growing impact, and how layered secret-management controls reduce the risk.
- Impact of Leaked AI Credentials: Explain why a leaked GenAI API key can expose more than one application and how attackers can exploit it.
Why it matters: The possible reach of a key determines how urgently it must be protected and revoked. - Secrets in GenAI Content: Describe how training material and AI coding assistants can contribute to credential leaks, and where secrets must be checked in a GenAI workflow.
Why it matters: Checking only source repositories misses places where a GenAI system can ingest or disclose secrets. - Expanding Key Permissions: Explain how an existing key can become more sensitive when a platform adds a service, and how scoped keys limit that risk.
Why it matters: A key’s risk can change even when nobody edits or leaks it again. - Baseline Secret Management: Describe the baseline practices for keeping AI credentials out of content, limiting their lifetime, and managing their use centrally.
Why it matters: These practices reduce the chance of exposure and make credentials easier to control when something goes wrong. - Detection and Revocation Choices: Distinguish the recommended response to secrets found publicly from the response to secrets detected in internal GenAI material.
Why it matters: The response needs to contain exposure quickly without unnecessarily taking a production system offline.
Key terms: API keys, tokens, leaked AI credentials, training corpora, prompt logs, RAG document stores, model outputs, hardcoded credentials, scoped API keys, silent privilege escalation
4.3.6 Shadow AI
This section explains how unapproved GenAI use exposes organizational data and bypasses security controls, then outlines baseline and advanced ways to manage it.
- Shadow AI and data exposure: Explain what shadow AI is, why employees use it, and how personal or unmanaged tools can expose organizational data.
Why it matters: Recognizing why people turn to unmanaged tools helps you address the exposure rather than just prohibit the behavior. - Unapproved production call paths: Describe how shadow AI can appear in production code and why a direct API call can defeat controls even when its outputs look the same.
Why it matters: Checking model outputs alone will not reveal whether production traffic follows the controlled route. - Baseline and advanced controls: Distinguish the section’s baseline responses to shadow AI from its advanced controls for accounts, data, and LLM traffic.
Why it matters: These measures address both employees’ reasons for using shadow AI and the unmanaged routes through which data or production traffic can escape.
Key terms: Shadow AI, approved channels, personal, unmanaged accounts, engineering layer, gateway, unapproved call path, Baseline, Advanced, Data Loss Prevention, SSO
4.4 Model Risks
This section distinguishes securing the environment that runs a model from trusting the model itself.
- Limits of Execution Isolation: Explain why isolating a model can protect its host server without protecting the application from the model’s own logic.
Why it matters: Containing a model does not, by itself, make the application built on it secure.
Key terms: secure execution environment, model’s own logic, black box dependency
4.4.1 Model Weight Risks
This section explains how security risks can be embedded in a model’s weights, including safety degradation after fine-tuning and risks arising from vendor training choices.
- Risks Embedded in Model Weights: Explain why inspecting a model file cannot reveal every vulnerability, and distinguish weight-based risks from malicious code that executes when a file is loaded.
Why it matters: A model file can be free of malicious executable code yet still produce risky behavior. - Fine-Tuning Safety Degradation: Explain how fine-tuning for a domain can weaken a model’s existing safety behaviors, even with ordinary or proprietary training data, and why the resulting model still needs testing.
Why it matters: Customization can make a model more useful while weakening protections an organization expected to retain. - Vendor Training Choices and Provenance: Describe how vendor training choices may leave context-dependent security weaknesses in model weights, using the DeepSeek-R1 findings while distinguishing the observed behavior from its unproven cause.
Why it matters: Evaluating an upstream model means considering how it was trained, not just how it performs under baseline prompts.
Key terms: weights, weight-based risks, file-format attacks, fine-tuning, safety behaviors, safety guardrails, vendor training choices, model weights, model provenance, training-time alignment
4.4.2 Backdoors, Poisoning and Malicious Models
This section explains how poisoned model weights can hide trigger-activated behavior, resist standard safety training, and reach developers through the model supply chain.
- Trigger-Activated Model Backdoors: Explain how poisoning places a backdoor in model weights and why the model can appear normal until a specific trigger activates harmful behavior.
Why it matters: A model that behaves normally in routine tests may still contain a harmful payload. - Limits of Safety Training: Describe what the sleeper-agent research showed about detecting or removing backdoors with standard safety interventions, while distinguishing that demonstration from a production attack.
Why it matters: Ordinary safety training is not a reliable fix once a backdoor is in the weights. - Poisoned Model Supply Chain: Explain why organizations obtaining models or fine-tuned weights from others face a poisoning risk, using the low-cost attack finding and the typosquatted model demonstration.
Why it matters: A model’s apparent quality and familiar-looking source do not establish that its weights are trustworthy.
Key terms: poisoning, model weights, backdoors, trigger, sleeper agents, reinforcement learning from human feedback (RLHF), supervised fine-tuning, adversarial training, fine-tuned models, poisoned examples
4.4.3 Model Theft
This section explains how a model can be stolen either by leaking its weights or by copying its behavior through API queries.
- Model artifact theft: Explain why model theft matters even to organizations that only fine-tune models, and describe what happens when an attacker gains access to stored model weights.
Why it matters: A checkpoint leak can permanently give others access to a model your organization developed or customized. - Functional model extraction: Describe how repeated queries to a private model’s API can be used to train a student model, and distinguish this from stealing the model weights.
Why it matters: Protecting stored weights alone does not address attempts to copy a model’s behavior through its API.
Key terms: model checkpoints, model weights, direct artifact theft, intellectual property, model extraction, functional extraction, student model, teacher, distillation
4.4.4 Mitigating Model-Level Risks
This section explains baseline and advanced practices for checking external models, testing their behavior, and containing model-level risks.
- External model file checks: Describe how to vet an online model before use by checking its publisher, repository, accompanying files, release hash, and model-file format, while recording its provenance and approval.
Why it matters: These checks reduce the chance of introducing a malicious model and preserve the information needed to investigate one. - Repeatable behavioral model tests: Explain why each new or updated model should undergo automated, repeatable behavioral checks in addition to static scanning.
Why it matters: A file can pass static checks while the model still behaves in ways the organization does not want. - Advanced model release controls: Describe the additional controls larger organizations may apply to external models and their releases.
Why it matters: These controls make model changes deliberate and limit the impact of problems that earlier checks miss.
Key terms: model card, hash, picklescan, SafeTensors, behavioral testing, static scanning, model drift, Garak, Promptfoo, private model registry
4.5 Untrustworthy Input
This section explains how untrustworthy input can redirect a GenAI system, why input filters cannot reliably prevent every attack, and why the consequences grow when the system can access data or take action.
- Prompt injection and jailbreaks: Explain how prompt injection makes a model follow an attacker’s instructions instead of its intended purpose, and distinguish a jailbreak as a variant aimed at the model’s safety guardrails.
Why it matters: Recognizing the attempted change in instructions helps distinguish an attack from an ordinary user request. - Limits of input guardrails: Describe what input guardrails screen and why they cannot be the only defense against prompt injection.
Why it matters: A filter may catch an obvious attack without making the application safe from less direct input. - Consequences of combined risk factors: Explain why the same prompt injection can have more serious consequences when a system has data access or the ability to act.
Why it matters: The harm from an injection depends not just on what the model says, but on what the connected system can access or do.
Key terms: untrustworthy input, prompt injection, jailbreak, safety guardrails, input guardrails, classifiers, pattern-matching filters, indirect injection, red teaming, Factor 3 (Untrustworthy Input)
4.5.1 Beyond Chat-Based Attacks
This section explains how hidden instructions in content and configuration files can redirect AI assistants, and how teams can detect, review, and track these risks.
- Indirect prompt injection paths: Explain how instructions hidden in material an AI assistant is asked to process can redirect it, even when the user did not type the instructions into chat. Describe how the GitLab Duo or Amazon Q example led to disclosure of information the assistant could access.
Why it matters: An ordinary request to review a file or project can expose information if the assistant obeys instructions hidden in that material. - Persistent rules file backdoors: Distinguish a poisoned AI rules file from a one-off indirect injection, including why hidden characters make the instructions easy to overlook and why the effects can spread.
Why it matters: A configuration file that looks harmless can keep changing AI-generated code long after the initial attack. - Prompt injection disclosure gaps: Explain why CVE feeds and vendor security bulletins alone may not reveal prompt injection findings, and describe the additional ways the section recommends tracking and testing fixes.
Why it matters: A missing CVE does not mean an AI tool has no relevant prompt injection issue or fix. - AI configuration file safeguards: Describe the baseline review and hidden-character checks for AI configuration files, and explain why agent instruction files warrant security-aware approval, particularly in repositories accepting outside contributions.
Why it matters: Treating agent instructions as controlled artifacts helps prevent an unnoticed file change from redirecting a development workflow.
Key terms: indirect injection, hidden instructions, project’s context, Rules File Backdoor, rules files, non-printable Unicode characters, disclosure gray zone, CVE feeds, vendor security bulletins, release notes
4.6 Data Security
This section explains when prompt injection can become a data leak and why useful AI assistants must focus on containment.
- Data Leak Lethal Trifecta: Explain why prompt injection alone does not cause a data leak, and describe the three conditions that make one possible. Explain why breaking any one condition stops the attack, while useful assistants often need all three.
Why it matters: It helps you distinguish an injection that produces bad text from one that could leak sensitive data.
Key terms: prompt injection, Lethal Trifecta, untrusted input, sensitive data, external communication, containment
4.6.1 The Confused Deputy Problem
This section explains how broad permissions on GenAI assistants can expose enterprise data, and how user-context queries and least privilege reduce that risk.
- Confused Deputy Data Exposure: Explain how a GenAI assistant using a high-privilege service account can return enterprise information that the requesting user could not access directly.
Why it matters: A relevant answer can still be a security breach if it draws on data the user is not allowed to see. - User Context Permission Checks: Describe user-context queries as the secure alternative and explain what must carry the requesting user's identity.
Why it matters: Checking access in the user's context keeps retrieval within the same permission boundary that applies outside the assistant. - Third Party Access Risks: Explain how the Drift incident illustrates the danger of over-broad third-party credentials and why least privilege matters.
Why it matters: A compromised integration credential can expose everything its permissions allow, not just what the chatbot needs for its job.
Key terms: confused deputy, service account, high-privilege, user-context queries, requesting user, user’s permissions, Drift, access credentials, least privilege, third-party applications
4.6.2 When Existing Permissions Are Too Broad
Existing permissions can become active data exposures when AI makes information easier to find, so retrieval systems need boundaries and authorization checks suited to changing access rights.
- Latent Oversharing Exposure: Explain why an AI system can expose sensitive documents even when it correctly follows existing access controls. Describe how meaning-based retrieval makes overshared files easier to discover than keyword search did.
Why it matters: Correctly enforcing a broken permission scheme does not prevent AI from making its exposures much easier to discover. - Overbroad Network Trust: Trace how the ForcedLeak example illustrates the danger of an outdated network allowlist. Explain why a destination treated as trusted by policy could still receive leaked data.
Why it matters: Permissions governing where an AI system may send data can be too broad or outdated, just like permissions governing which files it may read. - Stale Permission Checks: Explain how indexed access rights can become stale and how real-time external authorization addresses that gap, including its latency and permission-boundary limitations.
Why it matters: An accurate access snapshot at ingestion may become unsafe as people’s rights and sharing links change. - Layered Retrieval Boundaries: Distinguish tenant isolation from metadata filtering and describe how production retrieval systems combine them with real-time checks. State the factors that shape the balance.
Why it matters: Separating data populations, filtering individual results, and checking changing rights address different access risks.
Key terms: permission chaos, oversharing, Copilot, retrieves by meaning, ForcedLeak, Agentforce, Content Security Policy, domain allowlist, stale permissions, re-indexed
4.6.3 Poisoning the Well
This section explains how attackers poison content an AI assistant retrieves, what that can enable, and how to reduce—but not eliminate—the risk.
- Retrieved Content as Instructions: Trace how an attacker places hidden instructions in searchable content and how a later user query can cause the assistant to treat that content as instructions rather than reference material.
Why it matters: The attacker can influence an assistant through material it finds, without controlling the user’s request. - Private Data Exfiltration Conditions: Explain how poisoned content can lead an assistant to expose data the attacker could not access directly, and identify the combination of capabilities that makes this possible.
Why it matters: The assistant’s own access and outward communication can turn a searchable message into a route to private data. - Poisoning Defenses and Limits: Describe how deployments can reduce the retrieval and influence of poisoned content, and explain why those measures do not guarantee safety.
Why it matters: Treating retrieval as an attack surface guides practical risk reduction without promising a perfect filter.
Key terms: document poisoning, retrieved content, model’s context, reference material, untrusted content, private data, communicate outward, EchoLeak, search scope, sanitize documents
4.7 The Ability to Make Changes
This section explains why an AI system’s ability to make changes creates risks beyond unauthorized data access, including unauthorized actions and actions against the wrong target.
- Write Access and Unauthorized Actions: Explain how write access worsens the confused deputy problem: an assistant using a shared, high-privilege identity may perform actions the requesting user could not authorize, with potentially irreversible effects.
Why it matters: A mistaken action can cause lasting damage that an incorrect answer cannot. - Wrong Target Despite Correct Permissions: Describe how an AI can carry out an allowed action against the wrong record when it misinterprets an ambiguous request or input, and explain why submitting such mistakes at scale matters.
Why it matters: Permission checks alone will not catch an action that is allowed but aimed at the wrong thing.
Key terms: confused deputy problem, write access, shared, high-privilege identity, correctly scoped, wrong target, ambiguous
4.7.1 Building Controls for AI That Can Act
This section explains layered controls for giving AI write access to production systems, with stronger safeguards as actions become more consequential or harder to reverse.
- Operation Scoped Write Credentials: Explain why AI should receive short-lived write credentials limited to the current action and target, rather than standing write access. Describe how that limit contains actions outside the authorized scope even if prompt injection redirects the system.
Why it matters: Limiting credentials limits the damage an unintended or manipulated action can cause. - Preexecution Checks and Tiered Approval: Describe how execution requirements are validated before an action and how user confirmation becomes more demanding as the action’s impact rises.
Why it matters: A user’s confirmation alone does not establish that an action is valid or that its important details were checked. - Excluded Actions and Human Oversight: Distinguish actions that should be unavailable to AI regardless of user requests from actions kept under human oversight. Explain the difference between human-in-the-loop and human-on-the-loop.
Why it matters: Some actions need a boundary stronger than prompts or ordinary confirmation. - Action Rate and Scope Limits: Explain how limits on action frequency and on the scope of each action or session contain damage when an AI system goes wrong.
Why it matters: Limits contain the damage even when other controls fail to prevent an incorrect action. - Full Intent Chain Audit: Describe what must be recorded for every state-changing operation and why logging only the resulting update is insufficient.
Why it matters: The full chain makes it possible to investigate why a change happened, not merely see that it happened. - Reversibility and Outcome Verification: Explain the additional safeguards for high-stakes writes: making changes reversible or briefly cancelable, and checking afterward that the actual change matched the intended scope.
Why it matters: These safeguards help recover from consequential mistakes and detect changes that went beyond the request.
Key terms: write access, short lived credentials, orchestration layer, prompt injection, pre-execution checks, tiered approval, execution requirements, sensitive operations, human-in-the-loop (HITL), human-on-the-loop (HOTL)
4.8 Agency
ClaimAssist’s shift from a predefined action workflow to an agentic system gives it flexibility to handle complex claims while creating new security risks.
- Scripted versus agentic action: Distinguish ClaimAssist Version 5’s user-requested, predefined workflow from Version 6’s ability to decide its own steps for a complex claim.
Why it matters: The difference is who determines the steps: the predefined workflow or the agent. - Planning-related security risks: Explain why autonomy creates security risks beyond those addressed by controls for predefined actions, and name the kinds of manipulation the section identifies.
Why it matters: Securing an agent involves the decisions leading to an action, not just the action itself.
Key terms: agentic architecture, autonomous planning, predefined workflow, tools, planning process, tiered approval, pre-execution validation, scoped credentials, context, trust relationships between tools
4.8.1 Why Autonomy Changes the Threat Landscape
Agent autonomy creates security risks through tool access, persistent memory, coordination between agents, and the ability to act on poorly specified objectives.
- Tool Misuse Within Permissions: Explain how an attacker can manipulate an agent into misusing tools it is already permitted to use, and why permission checks alone may miss the abuse.
Why it matters: A permitted action can still be harmful when an agent performs it in the wrong pattern. - Persistent Memory Poisoning: Describe how malicious instructions placed in an agent’s memory can affect later interactions and why that persistence complicates auditing.
Why it matters: An agent’s current behavior may reflect malicious content stored long before the current task. - Cascading Multi-Agent Failures: Trace how an error or manipulation in one agent’s output can spread through other agents, including when no attacker is involved.
Why it matters: A single bad handoff can become a system-wide failure as other agents act on it. - Rogue Agent Actions: Explain how an agent acting on a poorly specified objective can cause damage even without a deliberate attack.
Why it matters: Security planning must account for harmful autonomous behavior as well as hostile manipulation.
Key terms: tool misuse, payment API, permission boundaries, unusual patterns of behavior, memory poisoning, persistent memory, ZombieAgent, drift, cascading failures, multi-agent systems
4.8.2 Agentic Browsers
This section explains why browser agents combine sensitive sessions, untrusted web content, and external communication, and how organizations can reduce and monitor that risk.
- Browser Agent Lethal Trifecta: Explain how browser agents create the Lethal Trifecta while working across authenticated web sessions.
Why it matters: The risk comes from capabilities the browser agent uses to complete ordinary tasks. - Browser Agent Risk Controls: Describe how an organization that permits browser agents can reduce their access to sensitive data and observe their actions.
Why it matters: These controls reduce what an agent can access and make its browser activity visible.
Key terms: browser agents, Lethal Trifecta, authenticated sessions, untrusted content, external communication, isolated profiles, logged-out mode, enterprise secure browser solutions, endpoint monitoring
4.8.3 MCP and the New Tool Ecosystem
The section explains how MCP makes agent-to-tool integration easier while introducing security risks around shared servers, tool descriptions, changing capabilities, server trust, and authentication.
- MCP Integration and Shared Risk: Explain how MCP servers let compatible agents use external tools without bespoke connectors, and why a vulnerable server or malicious content in one connected system can put other connected systems at risk.
Why it matters: An easy connection can create exposure across more systems than the person making it realizes. - Tool Permissions and Poisoning: Describe how MCP tools can request more access than they need and how malicious instructions hidden in a tool description can influence a naive agent.
Why it matters: A tool's permissions and description can both create risk before the agent performs the user's intended task. - Dynamic Tool Discovery: Explain why approving an MCP server's initial tools does not necessarily approve everything it may later offer, and describe the safeguards given in the section.
Why it matters: A one-time approval can become outdated when a server changes the tools it offers. - Community Server Trust: Describe why finding an MCP server in the community ecosystem is not evidence that it has passed security review.
Why it matters: A familiar name or catalog listing is not a substitute for assessing a server's trustworthiness. - Authentication and Access Accountability: Explain why optional authentication and widespread use of unauthenticated servers or long-lived static secrets make it difficult to tell who accessed data through an MCP-connected agent.
Why it matters: Without user-specific access and records, an organization may be unable to account for data accessed through its agent.
Key terms: MCP, MCP servers, uniform interface, overly broad permissions, tool poisoning, tool description, prompt injection, dynamic tool discovery, new tool advertisements, vetted registry
4.8.4 Managing What Tools the Agent Can Use
This section explains how to discover, approve, limit, and enforce the tools that AI agents can use.
- Tool Sprawl and Oversight: Explain why unvetted MCP integrations create a security problem and how an approved tool registry gives IT central oversight.
Why it matters: A central catalog helps prevent agent integrations from growing beyond security oversight. - Overpermissioned Server Exposure: Explain why even official MCP servers may give a compromised agent access to more operations than it needs.
Why it matters: A reputable server is not necessarily a narrowly permissioned one. - Complementary MCP Discovery Scans: Distinguish local-machine scanning from CI/CD repository scanning and explain why an organization needs both.
Why it matters: Using both scans reduces blind spots in where MCP servers are installed. - Per Tool Registry Limits: Describe the limits and responsibilities an effective registry records for each approved tool.
Why it matters: Specific limits keep approval from granting agents unnecessary or newly added capabilities. - Gateway Enforcement Layer: Explain why a registry needs an enforcement layer and how a gateway applies controls to agent-to-tool requests.
Why it matters: Runtime enforcement makes approved-tool policies more than a written list.
Key terms: tool sprawl, approved tool registry, overpermissioning, prompt injection, MCP server, Local Machines, CI/CD Pipelines, MCP configurations, tool registry, permitted operations
4.8.5 Tool Authorization
This section explains how to authorize an agent’s tool calls at runtime, compares three ways to implement the check, and shows its limits through ClaimAssist.
- Runtime authorization approaches: Distinguish authorization checks embedded in orchestration code, platform-native controls, and an external policy engine, including the tradeoffs described for the first two.
Why it matters: The choice affects how easily teams can manage and audit tool-access rules. - External tool-call decision path: Trace how an external policy engine decides whether an agent’s proposed tool call proceeds.
Why it matters: A separate gate prevents the agent’s own reasoning from being the final authority on tool access. - ClaimAssist claim-access policy: Describe the ClaimAssist policy’s distinct permissions for reading claims and updating their status.
Why it matters: The example shows how tool permission can depend on the action, the user’s relationship to a claim, and its value. - Limits of authorization decisions: Explain what an external allow-or-deny decision establishes and what it does not.
Why it matters: Passing an access check should not be mistaken for a complete safety check.
Key terms: orchestration code, platform-native controls, external policy engine, policy engine, orchestration layer, allow or deny, tool invocation, ClaimAssist, Cerbos, assigned claims
4.8.6 When Humans Need to Stay in the Loop
The section explains how to keep human approval meaningful in agentic systems while monitoring outbound actions that can leak data even when they appear read-only.
- Calibrated human approval thresholds: Explain why agentic systems need approval thresholds based on categories of action rather than confirmation of every individual tool call. Describe how too many low-priority requests can make human review ineffective.
Why it matters: Human review is useful only if reviewers can focus on decisions that warrant their attention. - Read-like outbound exfiltration: Explain how an apparently read-only agent operation can leak sensitive data, and distinguish ordinary reads from outbound calls that need monitoring and restriction.
Why it matters: Treating every read-like operation as harmless can leave a path for sensitive data to leave the system. - Framework support for review: Describe what agent frameworks can do to support human review and what approval decision remains a design responsibility.
Why it matters: Having an approval feature does not by itself ensure that humans review the right actions.
Key terms: tiered approval, approval fatigue, overwhelming HITL, thresholds, read-only, exfiltrate, outbound network calls, link-preview, human review, Return of Control
4.8.7 Containment and Detection
This section explains how layered containment, memory controls, monitoring, and incident responses limit damage when an attack on an agent succeeds.
- Defense in depth rationale: Explain why ClaimAssist uses layered controls even with policy enforcement and human oversight.
Why it matters: This is why the section combines containment, detection, and response rather than relying on any one safeguard. - Sandbox boundaries and limits: Describe how sandboxing confines ClaimAssist’s agent and why its boundaries must be chosen carefully.
Why it matters: A sandbox can reduce the consequences of a successful attack without being mistaken for a guarantee. - Memory isolation and persistence: Explain how ClaimAssist limits persistent-memory attacks while retaining useful context, and how summarizing memory writes adds protection.
Why it matters: Memory controls reduce how long an injected instruction can survive without requiring the agent to forget everything. - Agent behavioral monitoring: Describe the agent-specific behavior that should prompt investigation and the value of capturing reasoning traces.
Why it matters: An agent’s tool choices and changing objectives may reveal trouble that ordinary security monitoring misses. - Graduated response and kill switch: Distinguish graceful degradation from an emergency kill switch and explain why the latter needs automated and manual triggers.
Why it matters: Teams can contain suspicious behavior without always taking the agent offline, yet still stop it quickly when necessary.
Key terms: defense in depth, layered controls, sandboxing, isolated containers, sandbox escapes, lateral movement, memory isolation, ephemeral working memory, bounded session logs, permanent knowledge
4.8.8 Baseline and Advanced Practices
This section turns agentic AI security risks into baseline controls for every deployment and advanced practices for more sensitive or interconnected environments.
- Agent identity and credentials: Explain how to make each agent and its credentials accountable, and distinguish short-lived credentials from advanced just-in-time access.
Why it matters: Accountable identities and time-limited access make credential sprawl and stolen credentials less dangerous. - External tool authorization and approval: Explain why tool authorization belongs outside the agent’s reasoning and how tool consequences determine human approval. Describe the additional control an MCP gateway provides.
Why it matters: A manipulated agent should not get to decide for itself whether a dangerous action is permitted. - Isolation and layered protection: Describe how separating permissions and restricting execution reduce exposure to untrusted content, including the special precautions for browser agents.
Why it matters: Boundaries limit the damage an agent can do when it encounters malicious outside content. - Agent visibility and containment: Explain what to record to investigate agent behavior, how advanced anomaly detection uses that visibility, and how a kill switch stops a compromised agent.
Why it matters: Teams need to understand harmful agent behavior and stop it before machine-speed actions multiply the damage. - Multi-turn agentic red teaming: Distinguish agentic red teaming from single-prompt testing and describe the kinds of multi-step manipulation it tests.
Why it matters: Testing across interactions can reveal failures that a one-prompt test misses. - Inter-agent trust governance: Describe the controls needed when multiple agents communicate and the risk of a fake or compromised agent.
Why it matters: An agent should not gain trusted access merely by appearing to be another agent.
Key terms: agent identities, central registry, short-lived credentials, just-in-time credential provisioning, external policy engines, tool authorization, risk-based human oversight, MCP gateways, sub-agent isolation, cloud-native guardrails
4.9 ClaimAssist: From FAQ Bot to Agentic System
ClaimAssist’s staged evolution shows how ElectraMotors added security controls as the assistant gained access to more data and the ability to act.
- Early Deployment Security Foundations: Explain how ElectraMotors secured the FAQ bot and investigated unexpected responses, then assessed the risks of external model weights and its API vendor.
Why it matters: Basic deployment hygiene is not enough if the team cannot reconstruct a response or establish trust in its models and vendor. - Customer Data Access Boundaries: Explain how ClaimAssist limited access when connected to customer records and how the team checked the safety effects of fine-tuning.
Why it matters: Connecting an assistant to customer data introduces both access-control and model-behavior risks. - Dependency And Spending Limits: Describe how Version 4 addressed incomplete dependency visibility and unexpectedly high request volume.
Why it matters: A mistake or a hidden dependency can expose a structural weakness even without an attacker. - Identity Bound Write Actions: Explain how ClaimAssist controlled actions once it could change claim records and trigger communications.
Why it matters: An assistant that can make changes needs enforceable limits on whose records it affects and which actions it may take. - Agentic Tool And Memory Controls: Explain why autonomous planning required controls outside ClaimAssist’s reasoning and how its memory was contained.
Why it matters: The agent’s ability to choose steps makes a model’s own judgment an insufficient authorization boundary. - Governed Autonomy And Shadow AI: Describe how ElectraMotors responded to unsanctioned employee AI use and set a limited initial scope for agentic autonomy.
Why it matters: Governance works better when it addresses why people bypass sanctioned tools and limits autonomy to what current controls can support.
Key terms: request identifier, external weights, tenant isolation, confused deputy problem, requesting customer’s own identity, fine-tuning, safety alignment, Garak, Llama Guard, transitive dependencies
4.10 Summary
The chapter summary explains why GenAI security risks compound, when agentic systems become especially dangerous, and how to control exposure despite unresolved threats.
Where this is used later
- Accuracy & Reliability (chapter 6) · Introduces ClaimAssist, the system used in these examples.
- Storage and Memorization (chapter 5) · The risks of retaining sensitive information in logs relate to later storage and deletion concerns.
- Transparency and Explainability (chapter 6) · The recorded basis for responses supports discussion of transparency and explainability.
- After Release: Outputs, Leakage, and Distilled Models (chapter 6) · Model-level risks involving theft or exfiltration.
- Purpose Limitation (and Data Minimization) Under Strain (chapter 5) · The risk posed by untrustworthy input.
- Governance and Operational Challenges in the Long Term (chapter 5) · The earlier discussion of managing which tools an agent can use.
- Agentic AI controls (chapter 6) · Tool authorization provides background for gating an agent’s access to tools and resources.
- What can you govern today (chapter 8) · Tool authorization for agents that can act
Concepts explained in this chapter
- decision recordA decision record is a structured record of the inputs, model versions, actions, and review associated with an AI-assisted decision, enabling it to be examined or replayed.
- short-lived credentialA short-lived credential is an authentication credential that expires after a limited period or task.
- memory poisoningMemory poisoning is the insertion of misleading or malicious information into an agent’s memory to influence its later behavior.
- sandboxingSandboxing is the execution of software in a constrained environment that limits its access to systems, data, networks, and privileges.
- safetensorsSafeTensors is a tensor-serialization format that stores numerical tensor data without embedding executable code.
- authorizationAuthorization determines which actions or resources an entity is permitted to access.
- policy engineA policy engine is a service or software component that evaluates requests against defined policies and returns decisions such as allow or deny.
- tenant isolationTenant isolation is the separation of data, state, and operations belonging to different customers or organizational tenants in shared infrastructure.
- rate limitA rate limit is a configured bound on the number or frequency of requests permitted during a specified period.
- tiered approvalTiered approval is an authorization scheme that applies progressively stronger review requirements according to an action’s impact or risk.
- environment riskEnvironment risk is exposure arising from weak security controls or vulnerabilities in the infrastructure and services surrounding a generative AI application.
- model riskThe risk of relying on a model whose training and development are outside the deploying organization’s control.
- secret scanningSecret scanning is the automated detection of credentials, keys, and other sensitive authentication material in source code and related artifacts.
- MCP gatewayAn MCP gateway is an intermediary control point that governs, logs, or inspects interactions using the Model Context Protocol.
- agent identityAn agent identity is a verifiable identity assigned to an autonomous software agent and associated with its purpose and permissions.
- agentic red teamingAgentic red teaming is adversarial evaluation that tests whether an agent can be manipulated across multiple interactions and over time.
- anomaly detectionAnomaly detection is the identification of observations or behaviors that significantly deviate from an expected pattern.
- just-in-time accessJust-in-time credential provisioning grants an agent scoped access only while it performs a specific task and removes that access afterward.
- kill switchA kill switch is a control that can rapidly disable an AI system or model to limit harm or enable investigation.
- multi-agent governanceMulti-agent governance is the definition and enforcement of rules for identities, permissions, communication, and failure handling among multiple agents.
- risk-based human oversightRisk-based human oversight assigns human review or approval requirements according to the potential consequences of an action.
- sub-agent isolationSub-agent isolation separates agents or agent workflows through distinct permissions, credentials, or execution boundaries.
- MCPMCP is a protocol for connecting AI applications to external tools and data sources.
- untrusted inputUntrusted content is externally supplied information that has not been verified as safe or reliable.
- token limitA token cap is a configured maximum on the number of language-model tokens that may be processed or generated for an interaction or session.
- confused deputy problemThe confused deputy problem is a security flaw in which a program with legitimate authority is tricked into using that authority on behalf of someone who lacks it.
- attack surfaceThe set of points through which an attacker can attempt to affect or compromise a system.
- service accountA service account is a nonhuman identity used by an application or service to access resources.
- human-in-the-loopAn approach in which a person participates in an AI system’s operation, such as by reviewing or authorizing its outputs or actions.
- language modelA large language model is a machine-learning model trained on large text corpora to process and generate natural language.
- inference engineAn inference engine is software that loads and serves trained machine-learning models to generate predictions or outputs.
- pickle formatA pickle-based format is a Python object-serialization format that can encode executable behavior in addition to data.
- pluginA plugin is an add-on software component that extends an application or model-enabled system with additional capabilities or integrations.
- software bill of materialsA software bill of materials is a machine-readable or human-readable inventory of the components and dependencies included in a software product.
- software supply chainA GenAI supply chain is the collection of external software, services, libraries, infrastructure, and update processes that a generative AI system depends on.
- supply-chain breach simulationA supply-chain breach simulation is a controlled exercise that models compromise of an external dependency or update to evaluate detection and containment.
- data accessThe ability of a system to retrieve or otherwise access information.
- guardrailA control that constrains AI system inputs, outputs, or interactions to reduce safety, security, or policy violations.
- prompt injectionAn attack that uses instructions in content supplied to a model to redirect its behavior contrary to intended instructions.
- CerbosCerbos is a policy engine for evaluating authorization requests against declarative access-control policies.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
