liveBook · Manning

Hallucination

As explained in Machines that Think

An AI hallucination is a generated claim or content that is false, unsupported, or fabricated while being presented as plausible.

A hallucination is a generated statement that sounds plausible but lacks reliable grounding in evidence, source material, or the information supplied in a request. It may be false or fabricated while remaining fluent and detailed. That surface coherence makes the failure harder to recognize than an obviously broken response. section 6.1.1

Language models generate text by predicting likely continuations from learned patterns and available context, not by checking each claim against an internal representation of truth. When relevant patterns are weak, sparse, or ambiguous, a plausible continuation may extend beyond the evidence. Evaluation and fine-tuning can reinforce this tendency when attempting an answer is rewarded more than declining under uncertainty. section 6.0 section 6.1.2

Figure 6.1 Models learn patterns from training data and extend them to new cases. In this example, animals associated with water are often labeled as fish. When no clear example is available, as with the dolphin, the model does not stop or admit uncertainty. It continues by extending the strongest available pattern, incorrectly classifying the dolphin as a fish rather than a mammal.

The animal-classification example illustrates how extending a familiar pattern can produce a plausible but incorrect answer. section 6.1.2

Weak premises can also invite fabrication. A question about Einstein’s main ideas on personal finance implies an established reference where none has been demonstrated. A model may supply familiar-sounding advice instead of challenging the premise. The response fits the wording of the question without establishing that its claims are true. section 6.1.3

Fabricated legal citations

In Mata v. Avianca, attorneys submitted a brief citing nonexistent judicial decisions generated by a language model. The citations and quotations looked plausible, but the rulings could not be produced when requested. The judge sanctioned the attorneys and their firm, illustrating why convincing formatting is not verification. section 6.1.4

Risk grows when unsupported claims enter professional workflows and are copied into other documents, where their uncertain origin becomes less visible. In specialized settings, a fluent answer can appear actionable while misstating rules or procedures; it should not be treated as a substitute for appropriate professional judgment. section 6.1.4 section 8.5.3

Reducing hallucinations requires more than asking for accuracy. Reliable external information, verification steps, uncertainty monitoring, and human oversight can make outputs more grounded and easier to evaluate. These measures can reduce errors but do not guarantee their elimination; reliability depends on the boundaries and checks of the surrounding system. section 6.2 section 6.2.3

Hallucination is a serious failure of reliability, but it does not by itself settle whether a system has intelligence. Human false memories offer a limited parallel: confidence can detach from truth, even though human memory and model generation work differently. section 9.5.2 section 9.6

How Machines that Think covers it

How deep it goes
Explains probabilistic generation, weak evidence, extrapolation, and answer incentives, then outlines mitigation without claiming guarantees.
Examples
Uses a dolphin misclassified as a fish, a weak Einstein finance premise, and fabricated citations in Mata v. Avianca.
What you can do after
Evaluate model claims by distinguishing fluent presentation from evidential support and recognizing when external verification or professional oversight is needed.
How it connects
Connects hallucination to grounding, next-token prediction, extrapolation, uncertainty monitoring, and the stochastic parrot critique.

Also explained in

Compare how 37 books explain hallucination →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.