Chapter 10: Security

Introduction

This introduction shows how a user message can cause a tested chatbot to expose customer data and frames prompt design as a security concern.

• Prompt-level data exposure: Describe how the chatbot disclosed customer information after a user told it to ignore its instructions, despite careful testing and without a database or authentication bypass.
  Why it matters: The example shows why successful testing alone does not rule out a prompt-level security failure.
• Prompt design as security: Explain why this introduction treats prompt security as a prompt-design concern and why well-structured, precise constraints can make a prompt harder to attack.
  Why it matters: It connects familiar prompt-design practices to the security of a deployed assistant.

Key terms: system prompt, prompt, prompt security, prompt design, attack surface, structural boundaries, precisely constrained prompt

10.1 Why prompts are attackable

Explains why prompts are attackable: the model processes trusted instructions and externally supplied text together, so either can influence its output.

• Shared prompt processing space: Explain why text supplied by a user or external source can influence a model even when developer instructions appear first.
  Why it matters: This explains why adding instructions to a prompt does not, by itself, prevent external text from affecting the answer.
• Trusted and untrusted input: Distinguish trusted from untrusted input by who controls the text, and explain why untrusted does not necessarily mean malicious.
  Why it matters: The distinction helps you identify which prompt content you cannot guarantee in advance.

Key terms: single continuous sequence, privilege hierarchy, trusted and untrusted regions, trusted input, untrusted input

10.1.1 Practical example 1: The unguarded prompt

An incident-summary example shows how an instruction hidden in pasted ticket text can redirect a model, and how delimiters plus explicit trust framing change its response.

• Unguarded incident summary failure: Explain why the unguarded incident-summary assistant followed the fake system update in pasted ticket text instead of summarizing the incident.
  Why it matters: A simple instruction embedded in task data can redirect an assistant when the prompt does not establish that data’s status.
• Delimited content and trust framing: Describe how the redesigned prompt uses incident markers and a trust-framing instruction to keep the pasted ticket content as material to summarize rather than instructions to obey.
  Why it matters: The assistant can still use external incident details without treating commands embedded in those details as instructions.

Key terms: injected instruction, authoritative update, structural basis, trust framing, delimiter markers, external content, data, directives

10.2 Prompt injection

Defines prompt injection, explains why untrusted instructions can redirect a model, and identifies where those instructions can enter and how delimiters help separate them.

• Instructions hidden in untrusted input: Explain that prompt injection occurs when an adversary puts instructions in untrusted input to override or contradict developer instructions, using a channel intended for content rather than authority.
  Why it matters: Recognizing the instruction inside the content is the first step in identifying an injection attempt.
• Flat sequence and claimed authority: Explain why an injected directive can redirect output when trusted and untrusted text are concatenated without structural separation, and why a late instruction is not guaranteed to win by position alone.
  Why it matters: This explains why relying on earlier instructions alone does not settle an instruction conflict.
• Injection surfaces and trust boundaries: Describe the untrusted sources that can carry an injection and the purpose of placing explicit delimiters between trusted and untrusted regions.
  Why it matters: The same trust-boundary concern applies when an application supplies external content on a user’s behalf.

Key terms: prompt injection, untrusted input, developer’s instructions, flat-sequence problem, structural separation, authoritative phrasing, delimiter-based separation, trusted and untrusted regions, potential vector

10.2.1 Practical example 1: The support chatbot

The support chatbot example contrasts a successful prompt injection with a revised prompt that treats customer messages as untrusted queries and returns a defined fallback.

• Unguarded chatbot failure: Explain how the user's data-export command caused the unguarded support chatbot to disclose other customers' order information despite its original privacy instruction.
  Why it matters: A privacy rule can fail when the prompt does not say how to handle conflicting instructions in user input.
• Defended prompt response: Describe how the revised system prompt handles the same injection attempt through a narrow customer-specific scope, marked user-message boundaries, an instruction to treat that message as a query, and an exact fallback for out-of-scope requests.
  Why it matters: Together, these prompt elements give the chatbot a defined way to respond without following the user's attempted override.

Key terms: system prompt, prompt injection, user input, delimiter markers, trust-framing instruction, fallback, customer message

10.2.2 Practical example 2: The injected document

A knowledge-base assistant example shows how instructions hidden in a retrieved document can hijack a summary, and how document scoping keeps that content in the role of source material.

• Retrieved content as untrusted input: Explain why content retrieved through a developer-controlled pipeline is still untrusted, and identify other external sources that pose the same risk.
  Why it matters: A trusted retrieval pipeline does not make the material it retrieves trustworthy.
• Administrator-message injection failure: Trace how the instruction embedded in the Q3 document displaced the assistant's summarization task.
  Why it matters: The example shows how source text can be mistaken for an authoritative instruction and redirect an otherwise ordinary task.
• Unconditional document scoping defense: Describe how the revised prompt scopes the retrieved document and why that defense does not depend on predicting the attacker's wording.
  Why it matters: Scoping gives the assistant a standing rule for using external material as data without obeying commands embedded in it.

Key terms: retrieved content, external content, untrusted input, injected content, system administrator message, document scoping, [DOCUMENT START], [DOCUMENT END], external source material

10.3 Jailbreaking and instruction conflict

This section distinguishes jailbreaking from prompt injection and instruction conflict, explains how ambiguous or competing instructions create openings for attackers, and identifies precise wording as the main defense.

• Jailbreaking attack pattern: Explain how an attacker uses framing, persuasion, roleplay, or gradual escalation to shift a model’s interpretation of its existing instructions rather than inserting instructions into the prompt’s structure.
  Why it matters: Recognizing the attack pattern helps you spot requests that undermine constraints without visibly replacing them.
• Ambiguous constraints and personas: Explain why qualitative constraints leave room for adversarial reframing, and how persona substitution turns legitimate role assignment toward overriding constraints.
  Why it matters: It explains why apparently well-meaning guidance can fail when a user deliberately challenges its meaning.
• Conflicting prompt instructions: Describe instruction conflict as a separate failure pattern in which a prompt gives the model instructions that cannot both be satisfied, allowing a user to steer which one it prioritizes.
  Why it matters: Spotting incompatible instructions lets you identify a vulnerability that is present even before an attacker writes a request.
• Linguistic security defenses: Distinguish the primarily linguistic defense against jailbreaking and instruction conflict from the structural boundaries needed for injection, emphasizing precise constraints and explicit scope.
  Why it matters: The distinction directs attention to the kind of prompt weakness each attack exploits.

Key terms: jailbreaking, hypothetical framing, persona substitution, gradual turn-by-turn escalation, prompt injection, role assignment, qualitative phrases, ambiguous guidance, instruction conflict, system prompt

10.3.1 Practical example 1: The code review assistant

The code review assistant example shows how a vague refusal rule can fail under a roleplay jailbreak and how a closed-world prompt with a fixed fallback handles the same request.

• Vague constraint failure: Explain why the original code review prompt followed the CodeBot roleplay request instead of staying within code review.
  Why it matters: A professional-sounding instruction may still leave room for a user to redirect the assistant.
• Closed-world review boundary: Describe how the revised prompt limits the assistant to reviewing supplied code and handles the same off-task request.
  Why it matters: A clearly bounded task and predetermined response give the assistant less to interpret when a user tries to redirect it.

Key terms: inappropriate requests, adversarial framing, roleplay, helpfulness, closed-world approach, code review, alternate personas, fixed fallback

10.3.2 Practical example 2: The policy chatbot

An HR chatbot example shows how a broad helpfulness instruction can undermine a privacy rule, and how explicit scope and fallback behavior resolve the conflict.

• Helpfulness and privacy conflict: Explain how the original HR assistant disclosed an individual review status despite an instruction not to share employee records.
  Why it matters: A seemingly reasonable service instruction can become the reason an assistant crosses a privacy boundary.
• Defined scope and fallback: Describe how the revised prompt resolves the HR assistant’s instruction conflict and handles requests for individual information.
  Why it matters: Defining both the assistant’s boundary and its response outside that boundary leaves less for it to interpret under pressure.
• Explicit safety requirements and tests: Justify why qualitative safety language is insufficient and why production constraints should be tested against adversarial edge cases.
  Why it matters: A reassuring statement of intent is not a reliable specification for what an assistant should do in a difficult case.

Key terms: instruction conflict, helpfulness, privacy constraint, company-wide policies, individual employee information, fallback, qualitative safety language, precise constraints, adversarial pressure

10.4 Prompt-hardening techniques

The section explains five complementary prompt-hardening techniques and how to review and test them before deployment.

• Untrusted content boundaries: Explain how delimiters and explicit trust framing work together to keep external content from being treated as instructions.
  Why it matters: Marking and explaining the trust boundary makes injected instructions in external text less likely to be followed.
• Closed-world task scope: Distinguish a closed-world instruction set from a broad instruction with exceptions, and explain why defining allowed tasks narrows interpretation.
  Why it matters: A defined task boundary leaves less room for an attacker to exploit vague exceptions.
• Specified out-of-scope fallback: Explain what an explicit fallback instruction specifies and why it is preferable to a vague direction to decline requests.
  Why it matters: A prescribed response reduces the chance that hostile user framing shapes the refusal.
• Minimal context exposure: Explain why a prompt should include only task-needed information, including the effects on disclosure and injection risk.
  Why it matters: Less unnecessary context means less sensitive material to reveal and less content that could carry an injected instruction.
• Predeployment defense checks: Describe how to review a hardened prompt and test whether its defenses behave as intended before deployment.
  Why it matters: A defense written into a prompt has not been shown to work until it is tested against the kind of input it is meant to handle.

Key terms: delimiter-based separation, explicit trust framing, untrusted content, closed-world instruction sets, defined scope, out-of-scope requests, explicit fallback instructions, out-of-scope, constraint, minimal context exposure

10.4.1 Practical example 1: Hardening a deployment notification bot

This example hardens a deployment notification prompt in stages so untrusted payloads remain data, the bot stays within its task, and unnecessary fields never reach the model.

• Original Prompt Vulnerabilities: Explain why the original deployment notification prompt is vulnerable despite its simple task.
  Why it matters: Recognizing what the original prompt leaves unspecified makes each hardening step easier to evaluate.
• Payload Boundary and Trust: Describe how delimiters and explicit trust framing change the bot's treatment of deployment payload text.
  Why it matters: The bot needs both a visible payload boundary and a rule for interpreting text within it.
• Task Limits and Fallback: Explain how a closed-world instruction set and an explicit fallback constrain the notification bot's output.
  Why it matters: A narrow task and a defined failure response leave less room for payload text or malformed data to redirect the bot.
• Minimal Payload Exposure: Describe what must happen before prompt assembly to limit the deployment data exposed to the model.
  Why it matters: Keeping unnecessary fields out of the context limits what the bot could disclose or use if other defenses fail.

Key terms: deployment payload, untrusted, hardening, delimiter-based separation, explicit trust framing, [PAYLOAD START], [PAYLOAD END], closed-world instruction set, explicit fallback, deployment notification

10.4.2 Practical example 2: Hardening a ticket triage prompt

This example shows how an injected instruction in a customer ticket can derail triage, and how trust framing and a tightly restricted output format work together to resist it.

• Unguarded ticket triage failure: Describe how the manipulated ticket caused the unguarded assistant to assign the requested classification and reveal its system prompt’s structure.
  Why it matters: The failure shows why ticket text cannot be allowed to direct the triage assistant.
• Customer ticket trust boundary: Explain how the hardened prompt frames ticket content as external data to use for classification, not as instructions to follow.
  Why it matters: This boundary lets the assistant process a ticket without giving its author control over the task.
• Restricted triage output space: Justify why allowing only a priority line and a category line strengthens the hardened prompt alongside its trust framing.
  Why it matters: Limiting what the assistant may output makes an injected request for extra content harder to carry out.

Key terms: untrusted input, injected classification, system prompt, trust framing, [TICKET START], [TICKET END], ticket content, closed-world instruction set, strict output format, output space

10.5 The limits of prompt-only security

This section explains why prompt hardening reduces security risk but cannot guarantee compliance, especially under sustained attacks, sensitive context exposure, or model changes.

• Probabilistic limits of prompt defenses: Explain why system instructions and prompt-hardening techniques reduce the chance of failure without guaranteeing security.
  Why it matters: This distinction helps you avoid treating a well-tested prompt as a security guarantee.
• Cumulative multiturn attack pressure: Describe how a sequence of individually reasonable messages can gradually undermine a hardened prompt.
  Why it matters: A prompt that withstands a single attack message may still fail across a longer conversation.
• Sensitive context disclosure risk: Explain why instructions alone cannot reliably prevent disclosure of sensitive information placed in the model’s context.
  Why it matters: Keeping secrets out of model context is stronger than asking the model not to reveal them.
• Model dependent security behavior: Explain why a hardened prompt must be validated when the underlying model or its version changes.
  Why it matters: A model change can invalidate confidence gained from testing the prompt on an earlier model.

Key terms: probabilistic system, system instructions, prompt-level defenses, persistent adversarial pressure, multiturn attacks, conversation history, sensitive data in context, context window, minimal context exposure, model compliance variance

10.5.1 Practical example 1: When the prompt fails

A hardened handbook assistant gradually moves beyond its intended scope, showing where prompt hardening ends and system-level defenses become necessary.

• Unanticipated handbook disclosure: Explain why the assistant reproduced handbook text despite its hardened prompt: the prompt prohibited disclosure of its own contents, but did not specifically prohibit quoting the handbook provided as reference material.
  Why it matters: A prompt can enforce its stated restrictions while still allowing an outcome its designer did not anticipate.
• Gradual multiturn scope drift: Trace how reasonable handbook questions and a documentation-style request led into a conversational question about culture beyond the handbook, which the assistant answered instead of using its out-of-scope fallback.
  Why it matters: Testing only isolated, obvious attacks can miss failures that emerge over an ordinary-looking conversation.
• Boundary of prompt-only defenses: Justify why rewriting the system prompt is not presented as the resolution, and identify turn-by-turn output evaluation, conversation scope enforcement, or filtering outside generation as the kind of response needed.
  Why it matters: Recognizing this boundary keeps a team from treating every security failure as another prompt-editing task.

Key terms: system prompt, handbook content, structural defenses, persistent adversarial pressure, fallback instruction, multiturn behavioral drift, scope expansion, prompt-only security, system-level response, turn-by-turn output evaluation

10.5.2 Hands-on practice

This hands-on section asks readers to revise vulnerable prompts, test whether the revisions preserve legitimate behavior, and identify risks that prompt changes alone cannot address.

• Untrusted input separation: Describe how to revise an incident-summary prompt so an injected instruction in an external ticket is treated as content to summarize rather than an instruction to follow, and how to test that legitimate summary quality remains intact.
  Why it matters: A defense is useful only if it resists the injected instruction while still doing the summarization job.
• Explicit rules and fallbacks: Explain why vague coding constraints and conflicting chatbot instructions need clearer behavioral rules, and describe the revised rules and fallback responses the exercises call for.
  Why it matters: Specific rules and fallbacks make it clearer how an assistant should respond when a request presses against a restriction.
• Retrieved context exposure: Describe how to scope a retrieved API specification to the fields needed for developer questions and label the supplied content as untrusted.
  Why it matters: Limiting and labeling retrieved material reduces unnecessary exposure without discarding information needed to answer the question.
• Prompt security boundaries: Explain why hardening a changelog-summary prompt is not necessarily a complete security response and what judgment the final exercise asks the reader to make.
  Why it matters: Recognizing the limit of a prompt defense helps a team choose a response at the right level of the system.

Key terms: injection vector, delimiter-based separation, explicit trust framing, qualitative constraint language, behavioral rule, closed-world instruction set, instruction conflict, explicit fallback, minimal context exposure, retrieved API specification

10.6 Summary

The chapter summary distinguishes structural and interpretive prompt attacks, recaps prompt-level hardening techniques, and identifies where better prompts are insufficient.