liveBook · Manning

Chapter 6: The reliability problem

What you will learn in chapter 6 of Machines that Think

This introduction frames LLM reliability as a problem of fluent probabilistic text generation that can create misplaced trust, and previews hallucinations, safeguards, jailbreaks, and context limits.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction frames LLM reliability as a problem of fluent probabilistic text generation that can create misplaced trust, and previews hallucinations, safeguards, jailbreaks, and context limits.

Key terms: predicts the next fragment of text, patterns learned from vast collections of human language, information available at that moment, stable store of facts, facts, instructions, examples, retrieved evidence, prior context, signals

6.1 Sorry, are you hallucinating?

This section introduces the reliability problem in language models: fluent, coherent output can lack grounding in verifiable facts because generation follows statistical continuation.

Key terms: fluency, grounding, verifiable information, reliable foundations, statistical continuation, generation, coherence, factual support, evidence, prediction

6.1.1 When fluency outruns grounding

This section defines hallucination as fluent but ungrounded language-model output and explains why its hidden nature makes external validation necessary.

Key terms: hallucination, grounding, verifiable facts, source material, internal representation of truth, coherence, informational support, appearance of competence, structural risk, external validation

6.1.2 Optimization without truth

This section explains why next-token training and answer-focused evaluation make plausible but unsupported outputs an expected feature of language models.

Key terms: next token, training objective, statistical alignment, external fact, plausible continuation, factual correctness, pattern match, well-established patterns, ambiguous, relative likelihood

6.1.3 Extrapolating beyond experience

This section explains why language models hallucinate when they extend learned patterns beyond the experience or evidence that can reliably constrain them.

Key terms: hallucinations, extrapolation, constraints, generation, training data, sparse patterns, regression model, correlations, causes, weakly grounded prompts

6.1.4 When fabrication becomes operational risk

This section shows how convincing LLM fabrications become operational and institutional risks when they enter professional workflows and spread without verification.

Key terms: hallucinations, operational risk, professional workflows, institutional use, Mata v. Avianca, legal brief, judicial decisions, verification, sanctioned, Air Canada

6.2 Only facts, please

This section introduces process-level ways to reduce hallucinations by grounding, constraining, and evaluating language-model outputs rather than trying to make models inherently truthful.

Key terms: hallucinations, generation process, underlying limitation, grounded, constrained, unsupported claims, inherently truthful

6.2.1 Constraining generation

This section explains how external evidence and multi-step generation procedures can constrain LLM outputs to reduce, but not guarantee the elimination of, hallucinations.

Key terms: hallucinations, plausible continuation, verified truth, constraining generation, external information, external context, source material, retrieval-based generation, prompt, learned parameters

6.2.2 Monitoring confidence

This section explains how token-level uncertainty during generation can flag hallucination risk without determining whether an answer is true.

Key terms: token-level entropy production rate, probability distribution, next tokens, uncertainty, hallucination detection, single generation pass, verification, hallucination, truth

6.2.3 Reliability without guarantees

This section explains why hallucinations cannot be fully eliminated and why dependable LLM use requires external system safeguards and clear operating boundaries.

Key terms: hallucinations, mitigation strategies, uncertainty, inherent risk, reliability, retrieval pipelines, verification steps, uncertainty monitoring, human oversight, well-defined boundaries

6.3 Breaking the rules

This introduction challenges the appearance that LLMs follow instructions as rules and frames their behavior as pattern-based generation rather than procedural rule-keeping.

Key terms: instruction-following, constraints, rule-based behavior, generative system, patterns, procedures

6.3.1 Rules without rule-keeping

This section explains that apparent instruction-following in language models comes from probabilistic pattern continuation rather than procedural rule execution, making compliance uncertain under ambiguous or conflicting cues.

Key terms: prompts, instruction-following, symbolic operations, internal rule engine, learned associations, training, pattern imitation, rule enforcement, constraints, compliance

6.3.2 The impact of small details

This section shows that semantically irrelevant prompt details such as formatting, wording, and tone can change language-model outputs and measured accuracy.

Key terms: prompt, contextual cues, learned distribution, statistical signals, generation, prompt sensitivity, surface variation, spurious features, prompt robustness, measured accuracy

6.3.3 The difficulty of improving prompts

This section explains why trial-and-error prompt improvement is unreliable when feedback is missing and equivalent wording can affect models differently.

Key terms: reliable feedback, objective reference point, prompt refinement, Prompting in the Dark, classification tasks, prompt sensitivity, semantically equivalent prompts, meaning-preserving prompt variations, worst prompt performance, rigid scoring methods

6.3.4 Tilting the balance

This section explains how jailbreaks manipulate competing statistical cues in prompts to push aligned language models from refusal toward restricted responses.

Key terms: helpfulness, accuracy, harmlessness, responsiveness, predictive process, statistical cues, jailbreak, safety guidelines, refusal, framing

6.3.5 Containing instability

This section explains how production systems contain language-model instability with layered safeguards while accepting that probabilistic generation cannot be made fully reliable.

Key terms: operational risks, multiple layers of constraint, probabilistic generation, reliability, input filtering, output filtering, external checkpoints, calibration, policy gaps, architectural constraints

6.4 A budget for memories

This section introduces the limited information available to an LLM at any given moment and the reliability risks that follow when needed details fall outside that available information.

Key terms: parameters, current input, available information, conversations, documents, instructions, external sources, absent, displaced

6.4.1 Where the answer comes from

This section explains that an LLM can generate from only its fixed parameter knowledge and the text in its current prompt, which sets strict limits on what can influence an answer.

Key terms: encoded knowledge, model’s parameters, weights, training, text provided at inference time, prompt, inference, next-token prediction, active token sequence, fixed parameters

6.4.2 A window, not a memory

This section explains that a language model processes a bounded prompt context at inference time, so apparent memory and continuity depend on what fits within its context window.

Key terms: parameters, inference time, prompt, context window, tokens, active input, next prediction, truncate, compress, long-context reasoning

6.4.3 Present but not decisive

This section explains why information visible in an LLM’s context window may still have little effect on its response because attention assigns unequal influence across positions.

Key terms: context window, attention mechanisms, representational weight, positional effects, context rot, positional bias, U-shaped pattern, Lost in the Middle, long context, prompt repetition

6.4.4 Borrowed memory

This section explains how retrieval-augmented generation retrieves relevant external text for a prompt, extending access to knowledge beyond a model’s context window without changing how generation works.

Key terms: retrieval-augmented generation, RAG, embedding, external knowledge base, distance metric, vector space, context window, external knowledge, high-ranking passages, indexing

6.4.5 Retrieval is not grounding

This section explains why retrieving relevant documents for a RAG system improves available context but cannot ensure that a model’s answer is grounded in, correctly uses, or remains traceable to that evidence.

Key terms: retrieval, grounding, RAG, context window, generated answer, retrieved evidence, unsupported continuation, attribution, extrapolation, groundedness

6.5 Summary

This summary explains why language models can be unreliable, how mitigation techniques help without guaranteeing correctness, and how probabilistic behavior and limited context shape their outputs.

Where this is used later

Concepts explained in this chapter

← Chapter 5: Back to humans Chapter 7: Alignment and bias →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.