Chapter 2: Hallucinations

Introduction

Introduces hallucinations as a major AI limitation and explains the author’s concern that improvements may reduce mistakes without eliminating them.

• Persistence of AI mistakes: Explain the author’s concern that AI improvements make problematic cases less common and AI more usable, but do not seem to eliminate hallucinations; each remaining mistake may be harder to fix.
  Why it matters: It explains why hallucinations remain worth considering even as AI improves.

Key terms: hallucinations, epic fails, problematic cases

2.1 What are hallucinations?

Defines AI hallucinations by three characteristics: incorrectness, confident presentation, and unpredictability.

• Three defining hallucination traits: Explain that an AI hallucination is an incorrect output presented as correct, without caveats, that users may encounter unexpectedly.
  Why it matters: These three traits explain why a hallucination can be both misleading and hard to anticipate.

Key terms: hallucinations, incorrect, confident, unpredictable

2.2 Made-up facts

This section shows that LLMs can invent convincing facts, even when they claim those facts are real or are built for specialized research.

• Plausible but nonexistent facts: Explain that a made-up-fact hallucination can look credible while referring to something that does not exist, such as a legal case or a book title and author.
  Why it matters: A convincing answer can still contain wholly invented facts.
• Legal research hallucination risks: Describe how fabricated legal citations caused harm in the court example and why legal-research products in the cited study could not be assumed hallucination-free.
  Why it matters: A specialized tool’s promise does not remove the need to check consequential claims.

Key terms: hallucinations, made-up facts, LLM-based products, hallucination free

2.3 Misinterpretation

This section shows that AI can hallucinate by misinterpreting real information, not just by making up facts.

• Misinterpretation of real information: Explain how AI can give a false or harmful answer even when the information it draws on exists, using the search examples or retrieval-augmented generation (RAG).
  Why it matters: Providing a source does not, by itself, ensure that an AI answer interprets it correctly.

Key terms: hallucinations, misinterpretation, retrieval-augmented generation (RAG)

2.4 Incorrect solutions to problems

This section shows how AI can give incorrect solutions to tasks, sometimes while assuring users that the result is correct, across image generation, logic puzzles, coding, and image description.

• Incorrect results with reassurance: Explain how an AI can fail an explicit task while presenting its result as correct, using the image-generation examples.
  Why it matters: A confident response does not establish that the AI followed the request.
• Brittle logic puzzle solutions: Describe the farmer-and-sheep error and explain what the section says about LLM performance on slight variations of simple logic problems.
  Why it matters: A familiar-looking solution may not fit the precise version of the problem being asked.
• Coding and image-description errors: Describe the incorrect outputs reported for coding questions and image-description tasks, including why coding errors may be easy to overlook.
  Why it matters: Fluent explanations and plausible-looking code or descriptions can conceal errors that matter in practice.

Key terms: hallucination, incorrect solutions to problems, simple logic problems, river-crossing puzzles, wrong logic, API, visual content

2.5 Overconfidence

This section explains that AI can present wrong answers with unwarranted confidence and may resist correcting them even after a user points out an error.

• Confident presentation of wrong answers: Explain how an AI model can present a hallucinated answer as correct without acknowledging uncertainty, sometimes using a persuasive or reassuring tone.
  Why it matters: A confident tone is not evidence that an AI answer is correct.
• Limits of follow-up correction: Describe what can happen when a user points out an AI model’s mistake in a follow-up prompt, including both successful correction and resistance to changing the answer.
  Why it matters: A request to reconsider an answer is not a guarantee that the error will be fixed.

Key terms: hallucinated outputs, overconfidence, follow-up prompt, stubbornness

2.6 Unpredictability

This section shows that LLM hallucinations can appear unpredictably, even when a question is merely rephrased or a simple prompt is rerun.

• Unpredictable hallucination examples: Explain how a small change in wording produced opposite answers to the 9.11-versus-9.9 question, and how rerunning a simple prompt made a fabricated tower name disappear.
  Why it matters: A response that looks reliable in one attempt may not reveal what the LLM will do on another.
• Limits of hallucination fixes: Describe why the author compares efforts to fix hallucinations to Whac-a-Mole.
  Why it matters: Fixing a known error does not make future outputs free of surprises.

Key terms: hallucinations, unpredictable, Eiffenstein Tower, fine-tuned, Whac-a-Mole

2.7 Why does AI hallucinate?

The section frames AI hallucinations as a deeper problem than a minor bug and explains why understanding their causes matters.

• Hallucinations beyond minor bugs: Explain why hallucinations should not be treated as bugs needing only a minor fix, and why understanding their causes is useful.
  Why it matters: This framing helps you approach hallucinations as a problem to understand and manage, not just a small defect to patch.

Key terms: hallucinations, causes

2.8 Inadequate world models

Explains how AI can learn from examples yet develop inadequate world models that fail in unfamiliar situations.

• Limits of learning from examples: Explain why examples may be enough to learn a narrow task such as reading license plates but not enough to supply all the knowledge needed for driving.
  Why it matters: It explains why doing well on familiar examples does not guarantee sound decisions in new situations.
• Shortcuts in incomplete world models: Describe how incomplete world models can produce failures when a problem or scene differs from familiar examples, using the section’s LLM, cow, or traffic-cone examples.
  Why it matters: It helps explain why a model that succeeds on typical cases may still fail when a familiar pattern changes.

Key terms: world model, examples, internal world model, hallucinations, shortcuts, memorization

2.9 World models: Theory vs. practice

The section contrasts the theoretical promise of learning comprehensive world models from examples with the practical limits of available training data.

• Theory and long-tail limits: Explain why abundant, varied examples could in theory produce a comprehensive world model, and why rare situations make that approach impractical in practice.
  Why it matters: This distinction explains why more examples alone may not cover the unusual situations a model encounters.
• Unusual book-title task: Describe how the book-title example illustrates the limits of relying on examples or on indirectly combining related things a model has learned.
  Why it matters: The example makes the gap between a plausible theory of learning and observed performance concrete.

Key terms: world models, learning by example, edge cases, long tail, book titles, lengthy words

2.10 Misaligned objectives

This section explains how the gap between an LLM’s training objective and a user’s need for truthful, useful answers can cause hallucinations, even when alignment techniques or RAG are used.

• Training and use objectives: Explain how predicting likely next tokens differs from producing factual text and correct solutions, and why that difference can produce a plausible hallucination.
  Why it matters: A convincing answer is not necessarily a factual one.
• Limits of goal alignment: Describe what manually labeled data was used to improve and what limitation remained.
  Why it matters: Alignment can help without making hallucinations disappear.
• RAG and answer checking: Explain why adding factual database text to a prompt does not by itself resolve the truthfulness problem, and how the section says RAG can still be useful.
  Why it matters: Access to factual material does not remove the need to verify an LLM’s answer.

Key terms: objective, next-token predictions, misalignment, hallucination, manually labeled data, align, wedge, RAG, double-check answers

2.11 Toy hallucination example: Price optimization

A toy pricing example shows how a model that predicts purchases accurately can hallucinate when used to evaluate prices it has not learned to assess.

• Price optimization with predictions: Describe how the company uses a purchase-prediction model to compare possible prices, and state how it estimates expected revenue.
  Why it matters: This is the step that turns a purchase prediction into a proposed pricing decision.
• Missing price variation in training: Explain why a model can predict purchases accurately yet give implausible probabilities when the price changes.
  Why it matters: Good predictions on familiar products do not establish that the model can evaluate alternative prices.
• Prediction and pricing objective mismatch: Explain how the model's training objective differs from its pricing use, why implausible price scenarios may go unpenalized, and the resulting business risk.
  Why it matters: A prediction model's outputs can look useful for business planning even when training never checked the hypothetical scenarios behind that plan.

Key terms: revenue-maximizing price, probability of buying, expected revenue, training data, insufficient world model, price and sales, misaligned objective, loss, hallucinated outputs, strategic business decisions

2.12 Will hallucinations go away?

This section explains why more training data or current LLM methods may not eliminate hallucinations, and why their uncertain future calls for practical ways to live with them.

• Limits of more training data: Explain why simply adding training data may not eliminate hallucinations, given the limited supply of public human text and the amount of data further improvements may require.
  Why it matters: More data may help, but this section gives reasons not to count on it as a complete fix.
• Limits of current task formulation: Describe why some researchers doubt that more data alone can fix hallucinations under current LLM methods, including LeCun’s proposed limitations.
  Why it matters: This explains why the section considers a different methodology, not just a bigger dataset, potentially necessary.
• Uncertain timeline and practical response: Explain why the section does not predict when hallucinations will disappear, and describe the practical response it recommends meanwhile.
  Why it matters: It turns uncertainty about a future fix into a present-day decision about where and how to use AI.

Key terms: training data, public human text data, training examples, autoregressive LLMs, world models, physical intuition, new methodology, hallucinations, mitigate

2.13 Mitigation

The section explains several ways to reduce hallucinations in LLM outputs and one way to flag outputs for validation.

• Training data and human feedback: Explain how curating training data and using human feedback can reduce hallucinations, and why the section does not present either as a complete, scalable solution.
  Why it matters: These approaches can help, but they do not make hallucinations a solved problem.
• Context added during training: Describe the research example in which article titles were added before sentences during training and state the reported effect.
  Why it matters: The example shows that changing how training text presents context can affect hallucinations.
• Step-by-step prompt examples: Explain how the section's chain-of-thought prompt demonstrates a task before posing a similar one, and describe its reported effect on accuracy.
  Why it matters: A user can change the prompt to give the model a worked pattern for a similar problem.
• Retrieved documents as evidence: Explain why supplying retrieved, relevant documents through RAG can reduce hallucinations.
  Why it matters: Relevant source text gives the model something more specific to rely on when answering.
• Coder and tester cooperation: Trace the coder-and-tester LLM example and explain both its possible benefit and its limitation.
  Why it matters: Having another model check an output may help, but the check itself is not guaranteed to be right.
• Low-probability output warning: Explain how output probabilities can help identify responses that warrant validation.
  Why it matters: A probability-based warning can direct attention to answers that need checking.

Key terms: curate the training data, reinforcement learning with human feedback, RLHF, training process, Wikipedia article, prompt engineering, chain-of-thought prompting, intermediate natural language reasoning steps, RAG, retrieved information

2.14 Hallucinations can kill a product

The section uses customer-service and self-driving-car failures to show how hallucinations can threaten sensitive products, especially when unusual situations resist one-off fixes.

• Customer service product risk: Explain how a chatbot’s hallucinated refund advice harmed an airline’s customer-service product.
  Why it matters: A mistaken answer in a customer-facing product can become a business and legal problem, not just a bad response.
• Autonomous driving edge cases: Describe how the Cruise and Waymo incidents illustrate dangerous AI failures in unusual driving situations.
  Why it matters: In safety-sensitive products, an unusual case can turn an AI error into physical harm and a threat to operations.
• Limits of specific patches: Explain why fixing a known self-driving failure does not resolve the section’s broader concern about hallucinations.
  Why it matters: Fixing a demonstrated failure is valuable, but it does not establish that a sensitive product is safe in situations nobody has anticipated.

Key terms: hallucinations, customer-service chatbot, edge case, training data, classified or tracked, predicted the future motion, ad hoc actions, patch, software update

2.15 Living with hallucinations

The section argues that people using or building AI should plan for hallucinations from the start because they may persist and matter more in some applications than others.

• Planning for persistent hallucinations: Explain why hallucinations should be considered from the start of AI use or product development, and distinguish applications where they are not a big problem from cases that call for early assessment, mitigation, and detection planning.
  Why it matters: Considering hallucinations early helps you judge whether they pose a problem for a particular use of AI.

Key terms: hallucinations, mitigation, detection strategies

2.16 Summary