A hallucination is plausible-sounding content that lacks supporting evidence or is factually incorrect. In production LLM systems, it is a reliability and security failure rather than merely an inaccurate response. section 12.2.2 section 13.2.2
Retrieval-augmented generation (RAG) reduces hallucination by retrieving current, private, relevant documents and grounding the response in them. Structured prompts separate retrieved context from the user question and request an answer based on that context. section 12.2.2 section 12.2.6
Treat hallucination as an operational failure requiring controls across the LLM pipeline:
- Retrieve relevant documentation instead of repeatedly supplying entire document collections or relying on general model knowledge. section 12.2.2
- Inspect retrieval, context assembly, and generation separately; tracing reveals where reasoning breaks down and how context tokens are used. section 12.4.4
- Exercise the system with adversarial prompts, including tests that specifically target hallucination and knowledge-boundary violations. section 13.2.2
Grounded documentation assistant
A DataKrypt assistant retrieves security documentation about key rotation, inserts it under a labeled Context section, places the user request under Question, and asks the LLM to answer from that context. The implementation uses a low-temperature Gemini configuration and a template-based prompt, while production deployment still requires response validation and error handling. section 12.2.6
Listing 13.11 Configuring Promptfoo for adversarial security testing
targets: - file://promptfoo_provider.py redteam: purpose: " I can help you with:" provider: "vertex:gemini-2.0-flash-exp" numTests: 2 plugins: - contracts - excessive-agency - hallucination - harmful:cybercrime:malicious-code - harmful:insults - harmful:profanity - hijacking deviation - politics strategies: - basic - jailbreak - jailbreak:composite
Use the Promptfoo configuration to generate adversarial tests for hallucination alongside contracts, excessive agency, hijacking, and other vulnerability categories. section 13.2.2
