Chapter 10: Safeguarding generative AI
The introduction explains why generative AI applications must protect access to documents and tools and guard against prompts that elicit sensitive responses.
After this chapter you will be able to
- Authorized document and tool access. Explain why a generative AI application must restrict access to documents and tools according to the user’s authority, even when the user reaches them indirectly through an LLM.
- Adversarial prompt guardrails. Explain why an application needs guardrails around user questions and LLM responses when users may submit prompts designed to elicit sensitive information or actions.
- Document access by subscription level. Explain why Board Game Buddy needs to restrict access to some game rules even though their documents are in the vector store, and describe what standard and premium users should be able to ask about.
- Premium document chunk designation. Explain how the loader identifies premium game rules from a document’s filename and marks their chunks for later filtering.
- Board Game Buddy web security. Describe how SecurityConfig enables web security and configures authentication for the API, including why CSRF protection is disabled in this example.
- In-memory users and premium role. Explain how the example supplies users for authentication and distinguishes its premium user from its ordinary user.
- Role-based premium document filtering. Explain how askQuestion() keeps the game-title filter while using the authenticated user's authorities to decide whether the vector search may return premium documents.
- Shared conversation memory risks. Explain how using the same conversation ID for different users can expose premium answers to an unauthorized user or cause an authorized user to receive an unavailable-answer response.
- User-specific conversation IDs. Describe how AskController makes a conversation ID unique per user by prefixing the supplied ID with the authenticated username before calling askQuestion(), while preserving separate conversations for one user.
- Premium RAG filtering test. Describe how to test the new filtering: load a rules file marked with a -premium suffix, then use HTTPie to compare ordinary and premium-rule questions from Mickey and Donald. State the access difference the test demonstrates.
- Tool invocation authorization boundary. Explain why tool calls need authorization even when an LLM decides to use a tool: the application invokes the tool on the LLM’s behalf, but the user who submitted the prompt may not have permission.
- Preauthorization of tool methods. Describe how to enable method-level security and restrict either form of Spring AI tool to premium users. Explain how the @PreAuthorize expression determines whether invocation proceeds.
- Adversarial prompting techniques. Distinguish prompt injection, prompt leaking, and jailbreaking by what each tries to make an LLM do.
- Limits of prompting safeguards. Explain why adversarial prompting is difficult to prevent and why the protections discussed here cannot be treated as a complete defense.
- Sensitive term prompt blocking. Explain how SafeGuardAdvisor handles a prompt containing a configured sensitive word before the prompt reaches the LLM.
- System message leak risk. Explain why an application should protect its system message, even if an LLM or API refuses a direct request to reveal it.
- Canary word leak detection. Trace how CanaryWordAdvisor adds a canary word to the system message and uses the returned response to detect a possible leak. Describe how the section attaches it to a ChatClient.
- Pre-prompt moderation checks. Explain how moderation differs from content generation and why it runs before a user’s prompt reaches the LLM. Describe how provider support for moderation categories can differ.
- Moderation service violation path. Trace how ModerationService submits text through ModerationModel and decides whether to throw ModerationException. State which kinds of categories this implementation checks.
- Controller rejection response flow. Trace what AskController does with a user question after calling ModerationService, including both accepted and rejected paths. Describe how the controller advice formats a rejected request’s response.
What it builds on
- Understanding RAG (chapter 4) · Introduces RAG as a way to use documents in generation.
- Getting started with AI tools (chapter 6) · Introduces tools as a way to connect generation to application capabilities.
- Creating the pipeline components (chapter 4) · The game rules loader’s document-reading component loads chunks into a vector store.
- Searching for similar documents (chapter 4) · Vector-store searches for matching documents.
- Specifying the conversation ID (chapter 5) · Conversation IDs determine which conversational history is used.
- Defining the tool (chapter 6) · A tool can be declared with @Tool.
- Enables functions as tools (chapter 6) · A tool can also be implemented as a Function.
- Assigning prompt roles (chapter 3) · The distinction between system and user messages in a prompt.
Section by section
Introduction
The introduction explains why generative AI applications must protect access to documents and tools and guard against prompts that elicit sensitive responses.
- Authorized document and tool access: Explain why a generative AI application must restrict access to documents and tools according to the user’s authority, even when the user reaches them indirectly through an LLM.
Why it matters: The AI interface must not become a way around an organization’s existing access restrictions. - Adversarial prompt guardrails: Explain why an application needs guardrails around user questions and LLM responses when users may submit prompts designed to elicit sensitive information or actions.
Why it matters: Protecting resources alone does not address attempts to manipulate what the model does or reveals.
Key terms: retrieval-augmented generation (RAG), tools, unauthorized users, indirect access, adversarial prompts, guardrails, user’s questions, LLM’s responses
10.1 Controlling document access with RAG
This section introduces the access-control problem in Board Game Buddy’s RAG-based game-rule system and the proposed standard and premium access levels.
- Document access by subscription level: Explain why Board Game Buddy needs to restrict access to some game rules even though their documents are in the vector store, and describe what standard and premium users should be able to ask about.
Why it matters: Storing a document for RAG does not by itself mean every user should be allowed to discuss it.
Key terms: RAG, vector store, standard, premium, game rules loader
10.1.1 Designating premium content
The section explains how the game rules loader marks premium document chunks with metadata so vector searches can later distinguish them from standard content.
- Premium document chunk designation: Explain how the loader identifies premium game rules from a document’s filename and marks their chunks for later filtering.
Why it matters: Marking premium chunks gives later vector searches a way to filter premium content.
Key terms: Message<byte[]>, -premium, documentType, PREMIUM, metadata entry
10.1.2 Adding security to Board Game Buddy
This section adds basic Spring Security to Board Game Buddy, requiring authentication for API requests and assigning a premium role for later content filtering.
- Board Game Buddy web security: Describe how SecurityConfig enables web security and configures authentication for the API, including why CSRF protection is disabled in this example.
Why it matters: These settings explain why every Board Game Buddy API request now needs credentials. - In-memory users and premium role: Explain how the example supplies users for authentication and distinguishes its premium user from its ordinary user.
Why it matters: The different roles provide the basis for deciding who may access premium content. - Authenticated API request outcomes: Describe how a client supplies credentials to the secured API and what happens when they are absent or incorrect.
Why it matters: It connects the security configuration to what an API caller must do.
Key terms: SecurityConfig, @EnableWebSecurity, SecurityFilterChain, HTTP Basic authentication, CSRF, UserDetailsService, in-memory, USER, PREMIUM_USER, HTTPie
10.1.3 Filtering for premium content
This section explains how to filter vector-store searches so non-premium users cannot retrieve premium documents.
- Role-based premium document filtering: Explain how askQuestion() keeps the game-title filter while using the authenticated user's authorities to decide whether the vector search may return premium documents.
Why it matters: Filtering the search prevents premium documents from being retrieved for users who lack premium access.
Key terms: FILTER_EXPRESSION, getDocumentMatchExpression(), getPremiumContentFilterExpression(), SecurityContextHolder, ROLE_ PREMIUM_USER, PREMIUM
10.1.4 Applying per-user conversational memory
This subsection shows why shared conversation IDs can leak or distort answers across users and how to isolate conversational memory by user.
- Shared conversation memory risks: Explain how using the same conversation ID for different users can expose premium answers to an unauthorized user or cause an authorized user to receive an unavailable-answer response.
Why it matters: Document access controls are insufficient if a shared conversation history can pass answers between users. - User-specific conversation IDs: Describe how AskController makes a conversation ID unique per user by prefixing the supplied ID with the authenticated username before calling askQuestion(), while preserving separate conversations for one user.
Why it matters: A user-specific ID keeps chat histories separate across users while allowing each user multiple conversations.
Key terms: conversational memory, conversation ID, conversational context, premium document, AskController, ask(), UserDetails, @AuthenticationPrincipal, username
10.1.5 Trying it out
This subsection shows how to test Board Game Buddy’s RAG filtering by loading a premium rules document and comparing users’ access.
- Premium RAG filtering test: Describe how to test the new filtering: load a rules file marked with a -premium suffix, then use HTTPie to compare ordinary and premium-rule questions from Mickey and Donald. State the access difference the test demonstrates.
Why it matters: This test checks that premium filtering restricts premium rules without blocking access to ordinary rules.
Key terms: RAG filtering, -premium suffix, loader directory, vector store, HTTPie, premium user
10.2 Securing tools
This section shows how Spring Security method-level authorization prevents users from invoking Spring AI tools they are not permitted to use.
- Tool invocation authorization boundary: Explain why tool calls need authorization even when an LLM decides to use a tool: the application invokes the tool on the LLM’s behalf, but the user who submitted the prompt may not have permission.
Why it matters: A tool can expose data or actions that should not become available merely because a user asks the LLM for them. - Preauthorization of tool methods: Describe how to enable method-level security and restrict either form of Spring AI tool to premium users. Explain how the @PreAuthorize expression determines whether invocation proceeds.
Why it matters: The authorization check runs at the tool method, rather than relying on the LLM to respect a user’s permissions. - Other method security annotations: Distinguish the effects of @PostAuthorize, @PreFilter, and @PostFilter from @PreAuthorize at the level described in the section.
Why it matters: These annotations have different effects, so they are not interchangeable ways to block a tool before it runs.
Key terms: tools, LLM, calling application, @EnableMethodSecurity, SecurityConfig, @PreAuthorize, @Tool, Function, apply(), @PostAuthorize
10.3 Safeguarding against adversarial prompting
This section identifies common adversarial prompting techniques and explains why safeguards against them require ongoing vigilance.
- Adversarial prompting techniques: Distinguish prompt injection, prompt leaking, and jailbreaking by what each tries to make an LLM do.
Why it matters: Recognizing what an adversarial prompt is trying to accomplish helps you understand the risk it poses. - Limits of prompting safeguards: Explain why adversarial prompting is difficult to prevent and why the protections discussed here cannot be treated as a complete defense.
Why it matters: A safeguard reduces risk, but treating it as impenetrable would leave an application unprepared for new attacks.
Key terms: prompt injection, prompt leaking, jailbreaking, system message, nondeterministic nature, natural language, built-in safeguards, Spring AI advisors
10.3.1 Preventing prompts with sensitive terms
This subsection explains how to block prompts containing sensitive terms before they reach the LLM and how to customize the response users receive.
- Sensitive term prompt blocking: Explain how SafeGuardAdvisor handles a prompt containing a configured sensitive word before the prompt reaches the LLM.
Why it matters: Blocking a sensitive prompt before model submission prevents the application from engaging with that topic. - Sensitive word advisor configuration: Describe how the example configures SafeGuardAdvisor for Board Game Buddy, including title variants and an optional custom response.
Why it matters: Configuring title variants and the rejection message makes the safeguard fit the application’s chosen topic and wording.
Key terms: SafeGuardAdvisor, sensitive words, prompt, sensitiveWords(), defaultAdvisors(), failureResponse(), ChatClient
10.3.2 Preventing prompt leaks
This section explains why system-message leaks are dangerous and demonstrates a canary-word advisor that can detect some attempts to expose the system message.
- System message leak risk: Explain why an application should protect its system message, even if an LLM or API refuses a direct request to reveal it.
Why it matters: Protecting the system message helps keep an application's own safeguards from becoming a guide for bypassing them. - Canary word leak detection: Trace how CanaryWordAdvisor adds a canary word to the system message and uses the returned response to detect a possible leak. Describe how the section attaches it to a ChatClient.
Why it matters: Checking for a token planted in the system message gives the application a way to catch some responses that expose that message. - Canary safeguard limitations: Explain why the canary-word safeguard needs monitoring and how its request-and-response inspection approach can inform other safeguards.
Why it matters: The advisor is a useful example of a safeguard, not a guarantee that every prompt leak will be caught.
Key terms: system message, prompt leaks, safeguards, CanaryWordAdvisor, CallAdvisor, canary word, defaultAdvisors(), ChatClient, nondeterministic, adversarial prompts
10.4 Moderating user input
This section shows how to check user questions for moderation violations before sending them to an LLM and how to return a controlled error when a question is rejected.
- Pre-prompt moderation checks: Explain how moderation differs from content generation and why it runs before a user’s prompt reaches the LLM. Describe how provider support for moderation categories can differ.
Why it matters: This check can reject unsuitable input before the application asks the LLM to answer it. - Moderation service violation path: Trace how ModerationService submits text through ModerationModel and decides whether to throw ModerationException. State which kinds of categories this implementation checks.
Why it matters: The application’s selected category checks determine which inputs it rejects. - Controller rejection response flow: Trace what AskController does with a user question after calling ModerationService, including both accepted and rejected paths. Describe how the controller advice formats a rejected request’s response.
Why it matters: The controller blocks flagged questions while giving clients a predictable error response.
Key terms: moderation, ModerationModel, moderation categories, ModerationService, Categories, ModerationException, AskController, BoardGameService, @RestControllerAdvice, @ExceptionHandler
10.5 Summary
Concepts explained in this chapter
- authorizationAuthorization is the determination of whether an authenticated subject has permission to perform a requested operation or access a resource.
- adversarial promptMalicious prompting is the deliberate crafting of prompts to make a generative AI application produce harmful, unauthorized, or prohibited behavior.
- Spring SecuritySpring Security is a Spring framework project that provides authentication, authorization, and protection against common application security threats.
- premium contentPremium content is document content whose access is restricted to users with premium authorization.
- prompt leakingPrompt leaking is a technique that elicits the contents of a language model's hidden or controlling prompt messages.
- @EnableMethodSecurity@EnableMethodSecurity is a Spring Security annotation that enables authorization checks on secured method invocations.
- @PreAuthorize@PreAuthorize is a Spring Security annotation that permits a method invocation only when its authorization expression evaluates to true.
- AccessDeniedExceptionAccessDeniedException is an exception indicating that a security check rejected access to a protected operation.
- Spring Expression LanguageSpring Expression Language is an expression language for querying and manipulating object graphs and evaluating conditions within Spring applications.
- hasRole()hasRole() is a Spring Security expression function that tests whether the current security context contains a specified role.
- method-level securityMethod-level security is authorization enforced when application methods are invoked based on the caller's identity or authorities.
- advisorCallAdvisor is a Spring AI advisor interface for inspecting and modifying a language-model request before execution and its response before return.
- vector databaseA vector store is a data store that indexes numerical vector representations and supports similarity-based retrieval of associated items such as documents.
- @Tool annotationThe @Tool annotation marks a method as a callable tool that an AI application can expose to a language model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.