A customer asks a chatbot a question and receives a confident answer with no basis in the company’s documents. That answer may sound helpful while being wrong. In a high-risk service, one such response can outweigh many fluent, correct interactions. section 6.4.1 section 1.0
A hallucination is plausible but incorrect or unverified content generated without adequate evidence. An LLM predicts a likely continuation of its prompt; it does not know whether that continuation is true. Outdated knowledge, missing context, and reliance on broad training data make unsupported answers a particular concern in question-answering. section 1.2.1 section 6.4.1
Reduce the opportunity for an unsupported answer rather than trusting fluency:
- Ground answers in relevant, trusted documents through retrieval-augmented generation (RAG). Prompt instructions can help, but supplied source data provides a stronger basis for an answer. section 6.4.1 section 1.2.2
- Set contextual bounds and decide where human review is needed. Generated dialogue drafts and intent-training suggestions can be checked before they reach users. section 1.2.3 section 7.0
- Test across multiple inputs before deployment and review behavior afterward. For extracted transcript values, check that every value actually appears in the transcript. section 1.2.3 section 12.3.2
- Limit what the LLM can control when it supports an action. Validate its choice through the surrounding system rather than letting it invent all parameters for an API call. section 10.2.2
Choosing a bank account
Suppose a user asks for the balance of their son’s account. The LLM can select an account ID from accounts already associated with that user, while the application supplies the user ID to the balance API. An invented account ID makes the call fail rather than exposing another customer’s balance. Selecting the wrong account belonging to the user remains possible, so test ambiguous requests and consider asking for clarification. section 10.2.2
Listing 10.11 Using an LLM to perform a search
<|instruction|>
You are supporting a digital assistant. A user is asking a question #1
about one of their bank accounts. Use the contextual information #1
provided to identify the bank account they are most likely asking about. #1
<|user|>
How much money is in my son’s account? #2
<|context|>
User Name: Bob
Accounts: [ {"id":12345, "type":"checking", "owners":["Bob","Jane"], #3
"opened":"12/25/2000"}, {"id":23456, "type":"saving", #3
"owners":["Bob","Jane"], "opened":"1/3/2005}, {"id":34567, #3
"type":"saving", "owners":["Bob","Jack"], "opened": "2/4/2024"}] #3
<|output|>
Account id: 34567 #4
The search prompt shows the bounded choice: the LLM receives a user’s account list and returns an account ID, rather than generating the complete balance request. section 10.2.2
No single guardrail proves that every response is accurate. Match oversight to the consequences of a mistake, verify outputs against available evidence, and keep assessing performance after launch. section 1.2.3 section 12.3.2
