liveBook · Manning

Hallucination

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An AI system's production of information that is unsupported, fabricated, or factually incorrect while being presented as credible.

A hallucination is a claim generated by an AI system that is unsupported by verifiable evidence or fabricated. It can sound fluent and confident, making the failure easy to miss. Models can learn the form of a convincing answer before they acquire its factual grounding. section 1.1.3 section 1.5

A fabricated study

An assistant asked about the “2023 Stanford Marshmallow Experiment on AI Ethics” might invent a summary, methodology, findings, and a researcher’s name. The details may sound plausible, but the study does not exist. section 1.1.3

Detection turns a free-form answer into individual claims, then checks each claim against evidence. A simple faithfulness measure is the number of supported claims divided by the total number of claims. Extracting implicit claims and deciding what counts as support require judgment, so the ratio depends on those choices. section 1.1.3 section 1.5

Figure 5.1 A flow chart for choosing a hallucination detection strategy. Closed-domain tasks can use grounding based on sources to verify claims. Open-domain tasks require self-consistency checks, or rely on external fact retrieval, or other factual alignment signals.

Choose a detection strategy based on whether a supplied source defines the allowable answer. section 5.1

Use evidence suited to the task:

  • Closed-domain: check whether claims are grounded in the supplied document or retrieved context. section 5.1
  • Open-domain: without a supplied grounding source, use signals such as external fact retrieval or self-consistency checks. section 5.1

Factuality here means support from verifiable evidence, rather than a judgment about an abstract or subjective notion of truth. Detection identifies unsupported claims; it does not by itself explain their cause or prevent them. section 5.1 section 5.4.5

Mitigation can operate at several levels:

  • Prompt engineering can instruct a model to use supplied documents or admit uncertainty, but models may ignore instructions. section 5.4.5
  • Retrieval-augmented generation supplies relevant context, but poor retrieval or claims beyond the retrieved documents can still produce hallucinations. section 5.4.5
  • Fine-tuning on verified data, reinforcement learning from human feedback with factuality rewards, and constitutional AI can encourage more faithful behavior. section 5.4.5

No mitigation eliminates hallucination. Continue detection and monitoring to see whether the chosen measures work and to catch failures before they erode user trust or cause deployment problems. section 5.4.5 section 1.0

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines hallucination, outlines claim verification and detection strategies, and briefly surveys mitigation without developing each method in detail.
Examples
Illustrates fabrication with a nonexistent Stanford AI ethics study and plausible invented research details.
What you can do after
Evaluate generated claims against evidence, select a detection approach for the task, and understand mitigation limitations.
How it connects
Connects hallucination to factuality, faithfulness, retrieval-augmented generation, fine-tuning, and reinforcement learning from human feedback.

Also explained in

Compare how 37 books explain hallucination →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.