A hallucination is an incorrect, misleading, or fabricated response from a language model. It can sound plausible: when information is missing, the model may still try to complete an answer. Poor-quality training data can also contribute. Treat factual answers as unverified unless you supplied or checked the facts. section 1.4.2 section 6.1.1
A missing fact becomes a wrong answer
Asked for the total number of columns across three temples, a chatbot supplied counts even though the provided text lacked the needed figures. One count was wrong: it gave 30 columns for the Temple of Hera I, rather than 50. A safer prompt says to use only the supplied text and admit when the answer is unknown. section 6.1.1
For document Q&A, retrieval-augmented generation (RAG) retrieves relevant document chunks and places them in the prompt. This grounding reduces reliance on pretrained knowledge, but retrieval alone is not a guarantee. Tell the model to answer only from retrieved context and to say “I do not know” when that context is insufficient. Guardrails and validators can help; high-stakes answers still warrant human review. section 1.4.2 section 6.3.2
def prompt_template(question, text):
return f'Use the following pieces of retrieved context to
answer the question. Only use the retrieved context to
answer the question. If you don\'t know the answer, or
the answer is not contained in the retrieved context,
just say that you don\'t know. Use three sentences
maximum and keep the answer concise. \nQuestion:
{question}\nContext: {text}. Remember: if you do not
know, just say: I do not know. Do not make up an
answer. For example do not say the three temples have
got a total of three columns. \nAnswer:'
This prompt template makes the fallback explicit: use only retrieved context, and do not invent an answer when it is missing. section 6.3.2
Match the safeguard to the task:
- For text-to-SQL, include the target schema and sample records in a few-shot prompt to reduce invented table or column names. section 10.3
- For tool-based agents, describe tools clearly and instruct the model to obtain facts through them. Inspect tool calls to check whether it searched before answering. section 11.8.2
Test the failure case, not just an easy question. A simple Q&A prompt produced different results across model runs: one model admitted that the temple figures were missing, while an earlier model supplied a wrong total. An explicit fallback makes the desired behavior clear without making every answer automatically reliable. section 6.3.2
