Model hallucination is the generation of plausible-sounding but incorrect or unsupported content. In search, it becomes especially risky when a foundation model answers factual questions without access to a trusted information source. section 1.0 section 1.1
The underlying problem is lossy compression: a foundation model captures broad knowledge but does not retain the original data exactly, so facts and concepts can be confused. Retrieval-augmented generation reduces this risk by retrieving current information from a search index or vector database and supplying it as context. section 1.1 section 1.0
Generative search makes hallucination a product and governance concern:
- Generated answers can replace existing documents or extracted answers with newly synthesized content. section 15.2
- Unsupported content can amplify political or commercial bias, censorship, propaganda, or dangerous advice. section 15.2
- Grounding generation in retrieved results provides a more reliable basis for accurate and up-to-date responses. section 1.0
A generated search response
A system might generate an itinerary for three days in Paris tailored to fine dining, historic buildings, museums, and avoiding crowds. The response is useful as synthesis, but its claims and recommendations require grounding when factual accuracy matters. section 15.2
