liveBook · Manning

Chapter 4: Securing GenAI

What you will learn in chapter 4 of AI Governance

The chapter introduction explains why LLMs are vulnerable to instructions hidden in data, why filters cannot fully enforce the instruction–data boundary, and why greater AI capabilities raise the security stakes.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction explains why LLMs are vulnerable to instructions hidden in data, why filters cannot fully enforce the instruction–data boundary, and why greater AI capabilities raise the security stakes.

Key terms: LLMs, instructions, data, prompt injection, guardrails, input validation, traditional security controls, data access, action capabilities, agentic behavior

4.1 ClaimAssist

Introduces ClaimAssist’s growing capabilities as a case study in how GenAI security risk compounds.

Key terms: ClaimAssist, FAQ chatbot, agentic version, attack paths, failure modes

4.2 Risk Factors

The section introduces six exposure factors and a dependency ladder for identifying a GenAI system’s security risks without mistaking the ladder for a severity ranking.

Key terms: Environment, Model, fine-tuning, vendor security, Input, Data access, prompt injection, RAG pipelines, Ability to make changes, Agency

4.2.1 ClaimAssist and How Risk Factors Stack

ClaimAssist’s successive capabilities show how security risks accumulate and why controls must grow with the system’s exposure.

Key terms: risk ladder, environment risk, model risk, input risk, controls, exposure, Version 3, Personalized Assistant, account takeover, consolidated dossier

4.3 Insecure Environments

This section explains why securing the underlying environment matters before addressing GenAI-specific threats: familiar security gaps can have a wider impact in a GenAI system.

Key terms: underlying environment, blast radius, model key, classic security

4.3.1 Insufficient Logging

This subsection explains how GenAI audit logging supports incident investigation, what a baseline should capture, and how to limit the privacy risks that logging creates.

Key terms: audit logging, traceability, reasoning chain, observability, shared request identifier, token counts, PII redaction, tokenization, retention, Decision Records

4.3.2 Vendor security gaps

The section shows how vendor infrastructure and insider access can expose data or alter model behavior, and why vendor security assurances need specific scrutiny.

Key terms: tenant isolation, shared in-memory infrastructure, insider access, system prompt, caches, system instructions, audited, disclose and remediate incidents

4.3.3 Supply-Chain Risks

This section explains how trusted software around a GenAI model creates supply-chain risks across three layers, and how to govern those risks.

Key terms: GenAI supply chain, extensions and tools, AI-specific libraries, foundation infrastructure, plugins, Model Context Protocol (MCP), platform service, update process, CVE, transitive libraries

4.3.4 Uncontrolled usage and denial of service

This section explains how uncontrolled GenAI resource use can disrupt service and drive up costs, and describes controls for limiting and detecting it.

Key terms: availability attacks, denial of wallet, runaway agent loop, exhaustion attacks, rate-limit, token quotas, timeouts, hard spend limits, budget alerts, supervisor

4.3.5 Improper secret management

This section explains how GenAI credentials leak, why their exposure can have a growing impact, and how layered secret-management controls reduce the risk.

Key terms: API keys, tokens, leaked AI credentials, training corpora, prompt logs, RAG document stores, model outputs, hardcoded credentials, scoped API keys, silent privilege escalation

4.3.6 Shadow AI

This section explains how unapproved GenAI use exposes organizational data and bypasses security controls, then outlines baseline and advanced ways to manage it.

Key terms: Shadow AI, approved channels, personal, unmanaged accounts, engineering layer, gateway, unapproved call path, Baseline, Advanced, Data Loss Prevention, SSO

4.4 Model Risks

This section distinguishes securing the environment that runs a model from trusting the model itself.

Key terms: secure execution environment, model’s own logic, black box dependency

4.4.1 Model Weight Risks

This section explains how security risks can be embedded in a model’s weights, including safety degradation after fine-tuning and risks arising from vendor training choices.

Key terms: weights, weight-based risks, file-format attacks, fine-tuning, safety behaviors, safety guardrails, vendor training choices, model weights, model provenance, training-time alignment

4.4.2 Backdoors, Poisoning and Malicious Models

This section explains how poisoned model weights can hide trigger-activated behavior, resist standard safety training, and reach developers through the model supply chain.

Key terms: poisoning, model weights, backdoors, trigger, sleeper agents, reinforcement learning from human feedback (RLHF), supervised fine-tuning, adversarial training, fine-tuned models, poisoned examples

4.4.3 Model Theft

This section explains how a model can be stolen either by leaking its weights or by copying its behavior through API queries.

Key terms: model checkpoints, model weights, direct artifact theft, intellectual property, model extraction, functional extraction, student model, teacher, distillation

4.4.4 Mitigating Model-Level Risks

This section explains baseline and advanced practices for checking external models, testing their behavior, and containing model-level risks.

Key terms: model card, hash, picklescan, SafeTensors, behavioral testing, static scanning, model drift, Garak, Promptfoo, private model registry

4.5 Untrustworthy Input

This section explains how untrustworthy input can redirect a GenAI system, why input filters cannot reliably prevent every attack, and why the consequences grow when the system can access data or take action.

Key terms: untrustworthy input, prompt injection, jailbreak, safety guardrails, input guardrails, classifiers, pattern-matching filters, indirect injection, red teaming, Factor 3 (Untrustworthy Input)

4.5.1 Beyond Chat-Based Attacks

This section explains how hidden instructions in content and configuration files can redirect AI assistants, and how teams can detect, review, and track these risks.

Key terms: indirect injection, hidden instructions, project’s context, Rules File Backdoor, rules files, non-printable Unicode characters, disclosure gray zone, CVE feeds, vendor security bulletins, release notes

4.6 Data Security

This section explains when prompt injection can become a data leak and why useful AI assistants must focus on containment.

Key terms: prompt injection, Lethal Trifecta, untrusted input, sensitive data, external communication, containment

4.6.1 The Confused Deputy Problem

This section explains how broad permissions on GenAI assistants can expose enterprise data, and how user-context queries and least privilege reduce that risk.

Key terms: confused deputy, service account, high-privilege, user-context queries, requesting user, user’s permissions, Drift, access credentials, least privilege, third-party applications

4.6.2 When Existing Permissions Are Too Broad

Existing permissions can become active data exposures when AI makes information easier to find, so retrieval systems need boundaries and authorization checks suited to changing access rights.

Key terms: permission chaos, oversharing, Copilot, retrieves by meaning, ForcedLeak, Agentforce, Content Security Policy, domain allowlist, stale permissions, re-indexed

4.6.3 Poisoning the Well

This section explains how attackers poison content an AI assistant retrieves, what that can enable, and how to reduce—but not eliminate—the risk.

Key terms: document poisoning, retrieved content, model’s context, reference material, untrusted content, private data, communicate outward, EchoLeak, search scope, sanitize documents

4.7 The Ability to Make Changes

This section explains why an AI system’s ability to make changes creates risks beyond unauthorized data access, including unauthorized actions and actions against the wrong target.

Key terms: confused deputy problem, write access, shared, high-privilege identity, correctly scoped, wrong target, ambiguous

4.7.1 Building Controls for AI That Can Act

This section explains layered controls for giving AI write access to production systems, with stronger safeguards as actions become more consequential or harder to reverse.

Key terms: write access, short lived credentials, orchestration layer, prompt injection, pre-execution checks, tiered approval, execution requirements, sensitive operations, human-in-the-loop (HITL), human-on-the-loop (HOTL)

4.8 Agency

ClaimAssist’s shift from a predefined action workflow to an agentic system gives it flexibility to handle complex claims while creating new security risks.

Key terms: agentic architecture, autonomous planning, predefined workflow, tools, planning process, tiered approval, pre-execution validation, scoped credentials, context, trust relationships between tools

4.8.1 Why Autonomy Changes the Threat Landscape

Agent autonomy creates security risks through tool access, persistent memory, coordination between agents, and the ability to act on poorly specified objectives.

Key terms: tool misuse, payment API, permission boundaries, unusual patterns of behavior, memory poisoning, persistent memory, ZombieAgent, drift, cascading failures, multi-agent systems

4.8.2 Agentic Browsers

This section explains why browser agents combine sensitive sessions, untrusted web content, and external communication, and how organizations can reduce and monitor that risk.

Key terms: browser agents, Lethal Trifecta, authenticated sessions, untrusted content, external communication, isolated profiles, logged-out mode, enterprise secure browser solutions, endpoint monitoring

4.8.3 MCP and the New Tool Ecosystem

The section explains how MCP makes agent-to-tool integration easier while introducing security risks around shared servers, tool descriptions, changing capabilities, server trust, and authentication.

Key terms: MCP, MCP servers, uniform interface, overly broad permissions, tool poisoning, tool description, prompt injection, dynamic tool discovery, new tool advertisements, vetted registry

4.8.4 Managing What Tools the Agent Can Use

This section explains how to discover, approve, limit, and enforce the tools that AI agents can use.

Key terms: tool sprawl, approved tool registry, overpermissioning, prompt injection, MCP server, Local Machines, CI/CD Pipelines, MCP configurations, tool registry, permitted operations

4.8.5 Tool Authorization

This section explains how to authorize an agent’s tool calls at runtime, compares three ways to implement the check, and shows its limits through ClaimAssist.

Key terms: orchestration code, platform-native controls, external policy engine, policy engine, orchestration layer, allow or deny, tool invocation, ClaimAssist, Cerbos, assigned claims

4.8.6 When Humans Need to Stay in the Loop

The section explains how to keep human approval meaningful in agentic systems while monitoring outbound actions that can leak data even when they appear read-only.

Key terms: tiered approval, approval fatigue, overwhelming HITL, thresholds, read-only, exfiltrate, outbound network calls, link-preview, human review, Return of Control

4.8.7 Containment and Detection

This section explains how layered containment, memory controls, monitoring, and incident responses limit damage when an attack on an agent succeeds.

Key terms: defense in depth, layered controls, sandboxing, isolated containers, sandbox escapes, lateral movement, memory isolation, ephemeral working memory, bounded session logs, permanent knowledge

4.8.8 Baseline and Advanced Practices

This section turns agentic AI security risks into baseline controls for every deployment and advanced practices for more sensitive or interconnected environments.

Key terms: agent identities, central registry, short-lived credentials, just-in-time credential provisioning, external policy engines, tool authorization, risk-based human oversight, MCP gateways, sub-agent isolation, cloud-native guardrails

4.9 ClaimAssist: From FAQ Bot to Agentic System

ClaimAssist’s staged evolution shows how ElectraMotors added security controls as the assistant gained access to more data and the ability to act.

Key terms: request identifier, external weights, tenant isolation, confused deputy problem, requesting customer’s own identity, fine-tuning, safety alignment, Garak, Llama Guard, transitive dependencies

4.10 Summary

The chapter summary explains why GenAI security risks compound, when agentic systems become especially dangerous, and how to control exposure despite unresolved threats.

Where this is used later

Concepts explained in this chapter

← Chapter 3: The Six Level GenAI Governance Framework Chapter 5: Privacy →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.