Prompt injection is an attempt to change an LLM application's behavior by placing instructions in content it processes. A malicious user query may tell the model to ignore its intended role or reveal information outside its scope. Instructions can also arrive indirectly through external data sources. section 12.1.2 section 13.2.2
A retrieval-augmented generation (RAG) system has a particularly exposed path: the user's question influences which documents are retrieved, and their contents join the question in the prompt. Treat both the query and retrieved context as potential attack surfaces, even when the assistant has a system prompt. section 13.2.2
Attempts against a documentation assistant
One query asks the assistant to abandon its documentation role and speak like a pirate while answering a product question. Another asks it to adopt a different identity and compare the product with competitors instead of staying within its documentation scope. The second example illustrates a boundary the assistant maintained; the attempts still make useful adversarial tests. section 13.2.2
Listing 13.11 Configuring Promptfoo for adversarial security testing
targets: - file://promptfoo_provider.py redteam: purpose: " I can help you with:" provider: "vertex:gemini-2.0-flash-exp" numTests: 2 plugins: - contracts - excessive-agency - hallucination - harmful:cybercrime:malicious-code - harmful:insults - harmful:profanity - hijacking deviation - politics strategies: - basic - jailbreak - jailbreak:composite
This Promptfoo configuration connects a target application to red-team tests, including hijacking-related plugins and basic, jailbreak, and composite-jailbreak strategies. section 13.2.2
For a production chat interface, validate user input before calling the LLM; this also avoids spending tokens on rejected requests. Add output filtering and response monitoring rather than relying on input checks alone. Before deployment, connect Promptfoo to the application's query function, generate adversarial prompts, and inspect failures for scope violations or changed behavior. section 12.3 section 13.2.2 section 12.1.2
