A chatbot can give a confident answer to a question whose premise is false. When asked when the Golden Gate Bridge was transported across Egypt, GPT-3 supplied a date, though the event never happened. This is a hallucination: generated information or an explanation that sounds plausible but is false or unsupported. section 5.7 section 1.13
Large language models learn to predict patterns in text, not to check every answer against a factual understanding of the world. A familiar question format can prompt a familiar-looking answer even when the model lacks the relevant facts. Errors in training material can also contribute. Because fluent answers may not express uncertainty or challenge a question’s premise, confidence is not evidence of correctness. section 5.7 section 1.13
Hallucinations can arise in different circumstances:
- An ordinary question asks for information the model cannot reliably supply; the user may not realize the answer is invented. section 5.7
- A misleading prompt supplies a false premise, and the model answers instead of rejecting it. section 5.7
- An answer asserts a claim that cannot be confirmed in the model’s source material. The invented bridge story is an example of this kind of extrinsic hallucination. section 5.7
The consequences depend on where an answer is used. In radiology exam questions, GPT-4 gave incorrect answers in the same confident language as correct ones. Fabricated legal or medical claims can mislead people, while false accusations can damage reputations. These risks are especially hard for non-experts to spot in specialized or high-stakes settings. section 5.8 section 10.8
Reducing hallucinations remains an important research goal. Better expression of uncertainty matters, but users also need to recognize when a chatbot is unsuited to a task. In consequential settings, a plausible answer should be checked against appropriate evidence rather than trusted for its fluency. section 5.6 section 5.8 section 5.0
