Chapter 9: When machines seem to think
The chapter introduces a balanced way to judge whether current AI can count as intelligent without treating either its failures or its impressive performance as decisive.
After this chapter you will be able to
- Limits of easy verdicts. Explain why neither AI's errors nor its fluent, useful outputs settle whether it thinks: humans also make mistakes, while similar-looking behavior can arise from very different mechanisms.
- Criteria for qualified intelligence. Describe the questions the chapter will use to assess machine intelligence: how much performance and repeated success count, what mechanisms reveal, and whether missing grounding, agency, experience, or selfhood matter. State the possibility under examination—that AI might have a real form of intelligence deeply unlike ours.
- Performance versus underlying mechanism. Explain why the duck-test shortcut is difficult to apply to large language models: they can give explanations, reasons, and corrections, but their fluent performance comes from statistical learning rather than perception, intention, or experience. Describe the open question of how much that difference should matter.
- Brittle and adaptive imitation. Distinguish imitation that fails when conditions change from performance that adapts to unfamiliar situations, using ELIZA and a chess engine as examples.
- Language models in the middle. Explain why the section places large language models beyond brittle imitation but does not conclude that their performance proves intelligence. Describe the open question of whether performance or the mechanism behind it should carry more weight.
- Intelligence as an inferred capacity. Explain why neither thought nor intelligence can be observed directly: a correct answer shows what a system did, not how it did it. Describe how fields such as psychology infer intelligence from patterns of learning, reasoning, problem solving, and adaptation.
- Language models’ outward signs. Describe the signs of intelligence language models display: they adapt responses to context, acquire capabilities through training, and organize information for useful problem solving. Explain why these signs prevent simply excluding them from performance-based definitions of intelligence without proving they think.
- The lived experience gap. Distinguish training data, a model’s current context, and tools from the continuous contact with the world through which humans and animals gain experience. Explain why this gap leaves current language models between outright dismissal and confident attribution of intelligence.
- Intelligence relative to expectations. Explain that we infer intelligence from performance relative to what we expect a particular person or species to be able to do, rather than from one universal standard. Describe how judgments change with age, context, or a species’ way of living.
- Machine intelligence and familiar bias. Explain why applying context-sensitive standards makes it harder to dismiss language models from discussions of intelligence: they perform demanding tasks flexibly, although their unfamiliar workings make people suspicious. Distinguish that argument from a claim that their performance proves human-like thought.
- Performance and reliable competence. Distinguish a convincing result from competence that holds across cases, and explain why flexible, repeatable performance matters. Use the master copyist to show how identical-looking results can be judged differently when authorship, intention, and context matter.
- Simulation and functional activity. Distinguish simulating an activity from actually performing it, using storms and arithmetic, and explain why functionalism makes a system’s abilities more relevant than its biological makeup when judging intelligence.
- Original and derived intentionality. Explain that even stable, intelligent-looking performance leaves open whether a model’s words refer to things through its own beliefs, goals, and commitment to meaning, or derive their meaning from human language, training data, and use.
- Reasoning within generation. Explain how a model that breaks a problem into steps, checks intermediate results, and delays its answer differs from one that answers directly. Describe why chain-of-thought and reasoning models sharpen the question of machine intelligence: each generated token can become part of the path toward the next.
- Token by token reasoning. Explain how chain-of-thought prompting asks a model to generate intermediate steps before its final answer, and why those steps can help with tasks whose parts depend on one another.
- Structured reasoning paths. Distinguish the section’s extensions of a single chain: self-consistency samples paths and favors a recurring answer; Tree of Thoughts branches and evaluates paths; Graph of Thoughts allows paths to merge or be revised; and STaR uses successful generated rationales in further training.
- Reasoning as integrated computation. Explain how reasoning models move intermediate steps from an on-demand chain-of-thought prompt into model design, training, and inference-time computation, allowing more work before an answer.
- Hidden intermediate work. Distinguish a reasoning model’s intermediate work from the explanation a user sees: the model may generate, evaluate, or revise more material than it reveals, and its final explanation may show only part of that work.
- Training and significance of deliberation. Describe how examples of useful steps and reward-based training can encourage decomposition, checking, revision, and delayed answers. Explain that these strategies can improve performance and resemble human deliberation without proving that the model thinks.
- Intermediate steps as scaffolding. Explain how chain-of-thought can improve performance by making earlier steps available to later ones, guiding possible continuations, and allowing partial errors to be noticed or corrected.
- Limits of visible reasoning. Distinguish a useful chain-of-thought from a faithful account of how the model reached its answer: the words may help solve the problem, explain an answer to a user, or rationalize an answer whose causes remain unclear.
- Conditions for apparent ability. Explain why a direct answer may hide an ability that appears when the same model is allowed to build intermediate structure, and relate this conditional appearance to the question of emergence.
- Complex behavior from simple parts. Explain why the abilities of artificial neural networks cannot be understood by examining their individual operations alone: many simple, learned relationships interact at scale to produce abilities such as generating language, solving problems, and adapting to unfamiliar contexts.
- Emergence in organized systems. Explain that an emergent property belongs to a system of interacting parts, even when no part has that capacity on its own. Describe how organization and interaction can produce higher-level behavior.
- Abilities appearing at scale. Describe how an LLM capability can appear absent in smaller models but become useful at a larger scale, sometimes after performance seemed close to guessing. Explain that this apparent change occurs even though the underlying process remains optimization followed by token generation.
- Measurement and emergence limits. Explain how an exact-answer benchmark can turn gradual improvement into an apparent sudden jump, whereas a smoother metric may show continuous progress. State that emergence is useful for identifying abilities visible at scale but does not, by itself, prove understanding, intention, or experience.
- Interacting conditions for emergence. Explain how scale, data, architecture, and training can jointly support emergent behavior without any one factor automatically producing reasoning or understanding. Describe how accumulated patterns interact to form abstract representations, whose coordination can make an ability visible after a threshold is crossed.
- Amplifiers of latent abilities. Explain that prompts, examples, tools, longer context windows, chain-of-thought, and reasoning models can make an existing latent capacity visible, stable, or usable without creating it from nothing.
- Mechanism and system level competence. Explain why describing a language model as a token predictor does not, by itself, settle what competence the trained system displays: its abilities depend on billions of learned relationships interacting across many steps.
- Emergence and the intelligence question. Distinguish evidence that a system can develop unexpected capacities from proof that it is intelligent. Explain why reliable, flexible abilities that transfer to unfamiliar cases would make imitation a less adequate description, without settling the question.
- Burden of the illusion argument. Explain why unexpected abilities emerging from simple mechanisms make token-by-token prediction insufficient grounds for dismissing machine intelligence, and what a stronger skeptical argument would have to show.
- Controlled puzzle test design. Explain how “The Illusion of Thinking” tested reasoning models with puzzle families whose complexity could increase while the underlying rules stayed constant, requiring valid move sequences rather than just final answers.
- Three complexity performance regimes. Describe the reported pattern: ordinary language models can do better at low complexity, reasoning models gain an advantage at medium complexity, and both collapse at high complexity; near collapse, reasoning-model effort can decline despite available token budget.
- Algorithmic control and evidential limits. Explain why failure at higher complexity—even when given the correct algorithm—suggests difficulty reliably executing a procedure, not just discovering one. Distinguish that skeptical interpretation from proof that machines cannot reason: the study’s conclusions depend on its models, tasks, and measures.
- Stochastic parrot critique. Explain why the stochastic parrot analogy challenges the idea that fluent language-model output demonstrates understanding: models learn relationships in human-produced text rather than through direct contact with the world.
- Accumulated knowledge versus adaptation. Distinguish crystallized intelligence from fluid intelligence and describe how ARC-AGI tests a model's ability to infer rules and adapt when familiar patterns offer little help.
- Hallucinations and missing grounding. Explain how the stochastic parrot critique connects hallucinations to generating statistically plausible text without reliably checking it against reality, and state the resulting question about what connection to the world understanding requires.
- Embodied knowledge and world models. Explain why critics distinguish describing the world from learning through bodily interaction with it, and why they question whether language-derived knowledge forms a stable world model.
- Borrowed agency and consequences. Distinguish conversational or tool-supported simulations of agency from the agency critics attribute to beings that pursue goals and face consequences that matter to them.
- Simulated self and lived perspective. Describe why critics distinguish a model’s first-person language or conversational persona from a continuing self, and state the section’s concluding challenge about a lived position from which the world matters.
- Contrasting learning paths. Explain how human learning through relatively few encounters, mistakes, and lasting revisions differs from language models’ data-intensive training before use. Describe the skeptical question this raises without claiming that the models’ abilities are unreal.
- Limits of learning during use. Describe why a deployed language model can respond flexibly to prompts and corrections within a conversation yet normally does not retain those corrections or change its underlying model. State the qualifications the section gives for memory systems, fine-tuning, and external learning loops.
- Uneven abilities and tokenization. Explain why training and representation can give language models an uneven profile: they may handle advanced tasks yet stumble on simple-looking ones. Use the strawberry letter-counting test to illustrate how tokens differ from individual characters, while noting that newer models can solve such tasks with character-level steps or code.
- Limits of missing features. Distinguish among missing features that expose practical limits, bear on responsibility or personhood, or merely show that AI takes a different route from humans. Explain why each critique must be assessed by what it actually rules out.
- Benchmark artifacts and reasoning limits. Distinguish failures caused by a benchmark’s design from evidence of a model’s reasoning limits, using Tower of Hanoi and River Crossing as examples.
- Cumulative case for intelligence. Explain the case for judging general intelligence through an accumulating range of demonstrated abilities, without requiring perfection, universality, human similarity, or superintelligence.
- Novelty beyond mere recombination. Explain why following rules and building on existing material do not rule out novel or cognitively meaningful contributions from language models.
- Transfer from token prediction. Describe transfer of learning as reuse of learned structure across tasks, and explain how the mathematics study supports the possibility that next-token prediction develops more than surface copying.
- Hallucination and cognitive absence. Distinguish the reliability problem posed by AI hallucinations from the claim that unreliable output proves cognition is absent, using the comparison with human false memory without equating their mechanisms.
- Embodiment as an intelligence test. Explain why lacking human-style bodily experience does not, by itself, rule out intelligence, using the section’s examples of an extraterrestrial communicating through signals and people with different sensory or movement abilities.
- Partial language-shaped world models. Explain why language training may produce functional expectations about the physical world without yielding an animal's action-tested world model, and how ARC-AGI-3 illustrates the remaining limitation.
- Agency versus intelligence. Distinguish intelligence—understanding, reasoning, planning, and problem solving—from agency, the pursuit of goals as one’s own. Explain why lacking self-originating goals rules out moral autonomy and responsibility but does not, by itself, rule out intelligence.
- Functional versus lived self. Describe the limited self-related information current AI can represent, such as its task, tools, limits, instructions, and conversational context. Distinguish this functional self-model from a continuous, lived human self, whose absence does not clearly rule out intelligence.
- Tool supported flexible competence. Explain why uneven abilities and reliance on tools do not automatically make machine intelligence fake: models can coordinate external resources, just as humans extend thought with aids. State the section’s proposed test—whether a system turns what it has learned into flexible competence, rather than whether it developed exactly as humans do.
- Qualified machine intelligence. Explain why the section treats current AI as a form of intelligence despite its uneven performance and dependence on training data, prompts, tools, and surrounding systems.
- Human and artificial thought. Distinguish the forms of cognition the section attributes to AI from human thought, and explain why its conclusion is neither that AI thinks just like us nor that its behavior is empty simulation.
What it builds on
- Turing’s test and its evolving meaning (chapter 1) · Turing’s emphasis on performance as a way to assess machines.
- Syntax, semantics, and the limits of imitation (chapter 1) · The question of whether convincing language use amounts to understanding.
- A window, not a memory (chapter 6) · The distinction between information in the current context and a model’s memory.
- Neurons as a web of mathematics (chapter 3) · Introduces artificial neurons as mathematical operations within a network.
- Large models through broad exposure (chapter 4) · Introduces broad exposure and scale in large models.
- When training becomes memorization (chapter 3) · Distinguishes memorizing training examples from generalizing to new ones.
- When fluency outruns grounding (chapter 6) · The problem of fluent output outrunning grounding.
- Grounding without causation (chapter 4) · The distinction between multimodal grounding and causal involvement with the world.
Section by section
Introduction
The chapter introduces a balanced way to judge whether current AI can count as intelligent without treating either its failures or its impressive performance as decisive.
- Limits of easy verdicts: Explain why neither AI's errors nor its fluent, useful outputs settle whether it thinks: humans also make mistakes, while similar-looking behavior can arise from very different mechanisms.
Why it matters: This keeps the discussion from confusing imperfect performance with no intelligence, or convincing performance with proven understanding. - Criteria for qualified intelligence: Describe the questions the chapter will use to assess machine intelligence: how much performance and repeated success count, what mechanisms reveal, and whether missing grounding, agency, experience, or selfhood matter. State the possibility under examination—that AI might have a real form of intelligence deeply unlike ours.
Why it matters: These questions provide a more precise basis for judging AI than asking whether it simply behaves like a human.
Key terms: hallucinations, bias, brittleness, grounding, convincing performance, observable performance, competence, surface imitation, mechanism, agency
9.1 When it walks like a duck
This section introduces the tension between judging an AI system by its convincing behavior and judging it by the mechanism that produces that behavior.
- Performance versus underlying mechanism: Explain why the duck-test shortcut is difficult to apply to large language models: they can give explanations, reasons, and corrections, but their fluent performance comes from statistical learning rather than perception, intention, or experience. Describe the open question of how much that difference should matter.
Why it matters: Convincing behavior is evidence worth considering, but it does not settle what kind of intelligence produced it.
Key terms: performance, statistical learning, perception, intention, experience
9.1.1 A convincing performance
This section contrasts brittle and adaptive imitation to explain why language models’ convincing performance raises, but does not settle, the question of machine intelligence.
- Brittle and adaptive imitation: Distinguish imitation that fails when conditions change from performance that adapts to unfamiliar situations, using ELIZA and a chess engine as examples.
Why it matters: The distinction helps separate a convincing appearance from an ability that holds up under pressure. - Language models in the middle: Explain why the section places large language models beyond brittle imitation but does not conclude that their performance proves intelligence. Describe the open question of whether performance or the mechanism behind it should carry more weight.
Why it matters: It lets you take model capabilities seriously without treating fluent behavior as a settled answer about intelligence.
Key terms: brittle imitation, adaptive imitation, ELIZA, chess engine, large language models, intelligence, lived experience, nonbiological mechanisms, Turing
9.1.2 A tricky definition
The section explains why intelligence must be inferred from performance and why language models’ observable abilities make their status difficult to settle.
- Intelligence as an inferred capacity: Explain why neither thought nor intelligence can be observed directly: a correct answer shows what a system did, not how it did it. Describe how fields such as psychology infer intelligence from patterns of learning, reasoning, problem solving, and adaptation.
Why it matters: This distinction keeps a convincing answer from becoming automatic proof of thought. - Language models’ outward signs: Describe the signs of intelligence language models display: they adapt responses to context, acquire capabilities through training, and organize information for useful problem solving. Explain why these signs prevent simply excluding them from performance-based definitions of intelligence without proving they think.
Why it matters: The models’ real capabilities must be accounted for even when their underlying processes remain uncertain. - The lived experience gap: Distinguish training data, a model’s current context, and tools from the continuous contact with the world through which humans and animals gain experience. Explain why this gap leaves current language models between outright dismissal and confident attribution of intelligence.
Why it matters: It identifies the key limitation that makes the models’ apparent intelligence hard to classify.
Key terms: intelligence, thought, patterns of performance, construct, psychometrics, observable performance, context, learned patterns, internal parameters, problem solving
9.1.3 Intelligence in others
The section argues that intelligence is judged relative to a system’s expected capacities, and that this matters when evaluating animal cognition and language models.
- Intelligence relative to expectations: Explain that we infer intelligence from performance relative to what we expect a particular person or species to be able to do, rather than from one universal standard. Describe how judgments change with age, context, or a species’ way of living.
Why it matters: The baseline you choose changes what a performance tells you about intelligence. - Mirror test and its limits: Describe how the mark-and-mirror test turns a question about animal self-recognition into an observable act, and explain why passing or failing it does not settle the broader question of self-awareness.
Why it matters: A test can provide evidence while still favoring certain bodies and ways of responding. - Machine intelligence and familiar bias: Explain why applying context-sensitive standards makes it harder to dismiss language models from discussions of intelligence: they perform demanding tasks flexibly, although their unfamiliar workings make people suspicious. Distinguish that argument from a claim that their performance proves human-like thought.
Why it matters: Calling a machine’s performance imitation does not, by itself, settle what that performance reveals.
Key terms: expected baseline, comparative cognition, species-specific capacities, mirror self-recognition, visible mark, bodily self-recognition, language models, flexibility, bias of familiarity, imitation
9.1.4 What performance is worth
This section weighs what flexible performance can show about machine intelligence while distinguishing outward success from competence, instantiation, and intentionality.
- Performance and reliable competence: Distinguish a convincing result from competence that holds across cases, and explain why flexible, repeatable performance matters. Use the master copyist to show how identical-looking results can be judged differently when authorship, intention, and context matter.
Why it matters: A good result is evidence of ability, but what it proves depends on what you are evaluating. - Simulation and functional activity: Distinguish simulating an activity from actually performing it, using storms and arithmetic, and explain why functionalism makes a system’s abilities more relevant than its biological makeup when judging intelligence.
Why it matters: This distinction explains why neither humanlike output nor nonbiological construction settles whether a machine thinks. - Original and derived intentionality: Explain that even stable, intelligent-looking performance leaves open whether a model’s words refer to things through its own beliefs, goals, and commitment to meaning, or derive their meaning from human language, training data, and use.
Why it matters: It marks a question about what a system’s answers mean that outward performance alone cannot resolve.
Key terms: performance, competence, master copyist, simulation, instantiation, functionalism, intentionality, original intentionality, derived intentionality
9.2 Thought behind chains
This introduction explains why step-by-step generation makes the question of whether language models think harder to answer.
- Reasoning within generation: Explain how a model that breaks a problem into steps, checks intermediate results, and delays its answer differs from one that answers directly. Describe why chain-of-thought and reasoning models sharpen the question of machine intelligence: each generated token can become part of the path toward the next.
Why it matters: The apparent reasoning is happening during generation, not just in the final answer.
Key terms: chain-of-thought, reasoning models, token, generation
9.2.1 Thinking token after token
This section explains how generating intermediate steps can improve a language model’s answers and how several methods extend that approach.
- Token by token reasoning: Explain how chain-of-thought prompting asks a model to generate intermediate steps before its final answer, and why those steps can help with tasks whose parts depend on one another.
Why it matters: It explains why a model can reach a better answer without changing its basic token-prediction process. - Structured reasoning paths: Distinguish the section’s extensions of a single chain: self-consistency samples paths and favors a recurring answer; Tree of Thoughts branches and evaluates paths; Graph of Thoughts allows paths to merge or be revised; and STaR uses successful generated rationales in further training.
Why it matters: These approaches show different ways to use intermediate material, rather than relying on one unexamined path to an answer.
Key terms: chain-of-thought, intermediate reasoning steps, context, token by token, self-consistency, Tree of Thoughts, Graph of Thoughts, STaR, Self-Taught Reasoner, rationales
9.2.2 A new member of the family
This section explains how reasoning models turn prompted step-by-step reasoning into trained, sometimes hidden computation, while cautioning that improved performance does not prove machine thought.
- Reasoning as integrated computation: Explain how reasoning models move intermediate steps from an on-demand chain-of-thought prompt into model design, training, and inference-time computation, allowing more work before an answer.
Why it matters: This distinction explains why a reasoning model is more than a model asked to show its work. - Hidden intermediate work: Distinguish a reasoning model’s intermediate work from the explanation a user sees: the model may generate, evaluate, or revise more material than it reveals, and its final explanation may show only part of that work.
Why it matters: A visible explanation is not necessarily a complete record of the computation behind an answer. - Training and significance of deliberation: Describe how examples of useful steps and reward-based training can encourage decomposition, checking, revision, and delayed answers. Explain that these strategies can improve performance and resemble human deliberation without proving that the model thinks.
Why it matters: The section links how reasoning behavior is learned to why its resemblance to thought is intriguing but inconclusive.
Key terms: reasoning models, chain of thought, inference-time computation, intermediate work, internal, hidden, compressed, final response, supervised fine-tuning, reward-based training
9.2.3 Lessons from reasoning
The section explains how giving a model room for intermediate steps can reveal reasoning-like performance, while cautioning that those steps do not necessarily reveal how the answer was produced.
- Intermediate steps as scaffolding: Explain how chain-of-thought can improve performance by making earlier steps available to later ones, guiding possible continuations, and allowing partial errors to be noticed or corrected.
Why it matters: It explains why changing how a model answers can change what it can solve. - Limits of visible reasoning: Distinguish a useful chain-of-thought from a faithful account of how the model reached its answer: the words may help solve the problem, explain an answer to a user, or rationalize an answer whose causes remain unclear.
Why it matters: A convincing written explanation is not, by itself, proof of the process that produced an answer. - Conditions for apparent ability: Explain why a direct answer may hide an ability that appears when the same model is allowed to build intermediate structure, and relate this conditional appearance to the question of emergence.
Why it matters: It cautions against judging a model’s abilities from one response format and introduces the question of when abilities appear.
Key terms: chain-of-thought, intermediate structure, context, partial results, explanation, computation, scaffold, misleading trace, reasoning-like behavior, reasoning mechanisms
9.3 More than the sum
The section introduces the question of how simple learned operations can, through large-scale interaction, produce complex behavior.
- Complex behavior from simple parts: Explain why the abilities of artificial neural networks cannot be understood by examining their individual operations alone: many simple, learned relationships interact at scale to produce abilities such as generating language, solving problems, and adapting to unfamiliar contexts.
Why it matters: This frames the question of how sophisticated behavior can arise from individually simple components.
Key terms: artificial neural networks, learned relationships, at scale
9.3.1 When ability appears
The section explains how abilities can become visible in larger language models, while cautioning that the appearance of a sudden new ability can depend on how performance is measured.
- Emergence in organized systems: Explain that an emergent property belongs to a system of interacting parts, even when no part has that capacity on its own. Describe how organization and interaction can produce higher-level behavior.
Why it matters: This gives you a way to discuss a system’s abilities without attributing them to each component. - Abilities appearing at scale: Describe how an LLM capability can appear absent in smaller models but become useful at a larger scale, sometimes after performance seemed close to guessing. Explain that this apparent change occurs even though the underlying process remains optimization followed by token generation.
Why it matters: It separates a visible change in performance from an assumption that the model gained a wholly new way of working. - Measurement and emergence limits: Explain how an exact-answer benchmark can turn gradual improvement into an apparent sudden jump, whereas a smoother metric may show continuous progress. State that emergence is useful for identifying abilities visible at scale but does not, by itself, prove understanding, intention, or experience.
Why it matters: It helps you avoid treating either a benchmark jump or an impressive ability as proof of a human-like mind.
Key terms: emergent property, complex systems science, local interactions, emergent abilities, scale, apparent discontinuity, token generation, metrics, benchmark, mirage
9.3.2 The recipe for emergence
This section explains how interacting training conditions can produce apparently sudden abilities, distinguishes that from grokking, and shows how usage conditions can reveal latent capacities.
- Interacting conditions for emergence: Explain how scale, data, architecture, and training can jointly support emergent behavior without any one factor automatically producing reasoning or understanding. Describe how accumulated patterns interact to form abstract representations, whose coordination can make an ability visible after a threshold is crossed.
Why it matters: It explains why an ability can appear suddenly even when the changes that support it have been gradual. - Delayed generalization in grokking: Distinguish grokking from emergence across model scale: within the same model, training examples can be learned early while performance on new examples remains poor, then generalization improves suddenly after much more training.
Why it matters: Grokking illustrates that fitting examples and learning a generalizable rule can happen at different times. - Amplifiers of latent abilities: Explain that prompts, examples, tools, longer context windows, chain-of-thought, and reasoning models can make an existing latent capacity visible, stable, or usable without creating it from nothing.
Why it matters: A model may show different abilities under different conditions of use, so an ordinary interaction may not reveal everything it can do.
Key terms: pattern accumulation, pattern interaction, intermediate representations, coordination across components, threshold dynamics, grokking, training loss, validation loss, generalization, amplifiers
9.3.3 From emergence to intelligence
The section argues that emergent abilities make token prediction an incomplete description of a language model’s competence, while leaving open whether that competence amounts to intelligence.
- Mechanism and system level competence: Explain why describing a language model as a token predictor does not, by itself, settle what competence the trained system displays: its abilities depend on billions of learned relationships interacting across many steps.
Why it matters: A description of how each output is generated may miss what the organized system can do. - Emergence and the intelligence question: Distinguish evidence that a system can develop unexpected capacities from proof that it is intelligent. Explain why reliable, flexible abilities that transfer to unfamiliar cases would make imitation a less adequate description, without settling the question.
Why it matters: Emergence changes what is worth asking about machine intelligence, but it does not answer the question on its own.
Key terms: token generation, learned relationships, emergent behavior, competence, emergence, imitation, reliable, flexible, transferable, biological cognition
9.4 The illusion hypothesis
The section reframes skepticism about machine intelligence: token-by-token prediction alone does not rule it out, so skeptics must identify an essential element that mechanism cannot provide.
- Burden of the illusion argument: Explain why unexpected abilities emerging from simple mechanisms make token-by-token prediction insufficient grounds for dismissing machine intelligence, and what a stronger skeptical argument would have to show.
Why it matters: This sets a clearer standard for judging claims that machine thinking is only an illusion.
Key terms: emergence, token-by-token prediction, essential to intelligence
9.4.1 The illusion of thinking
This section uses controlled puzzle tests to examine where reasoning models succeed, where they fail, and what those failures do—and do not—show about machine reasoning.
- Earlier evidence of fragile reasoning: Describe how earlier studies challenged claims of robust language-model reasoning through inconsistent problem solving, weak performance on unfamiliar abstract patterns, and sensitivity to superficial changes in math problems.
Why it matters: These studies show why the later puzzle experiment addressed an existing concern rather than starting the debate. - Controlled puzzle test design: Explain how “The Illusion of Thinking” tested reasoning models with puzzle families whose complexity could increase while the underlying rules stayed constant, requiring valid move sequences rather than just final answers.
Why it matters: Controlling the rules makes a breakdown at higher complexity more informative than a failure on unrelated hard questions. - Three complexity performance regimes: Describe the reported pattern: ordinary language models can do better at low complexity, reasoning models gain an advantage at medium complexity, and both collapse at high complexity; near collapse, reasoning-model effort can decline despite available token budget.
Why it matters: The change across complexity levels is the central result behind the paper’s challenge to reasoning claims. - Algorithmic control and evidential limits: Explain why failure at higher complexity—even when given the correct algorithm—suggests difficulty reliably executing a procedure, not just discovering one. Distinguish that skeptical interpretation from proof that machines cannot reason: the study’s conclusions depend on its models, tasks, and measures.
Why it matters: This distinction supports a critical reading of impressive reasoning traces without claiming the experiment settled machine cognition.
Key terms: GPT-4 Can’t Reason, Large Language Models Are Not Strong Abstract Reasoners, GSM-Symbolic, The Illusion of Thinking, Tower of Hanoi, Checker Jumping, River Crossing, Blocks World, initial state, three regimes
9.4.2 It’s just a parrot
This section explains the stochastic parrot critique of language models, using unfamiliar-task benchmarks and hallucinations to question whether fluent language is grounded in understanding.
- Stochastic parrot critique: Explain why the stochastic parrot analogy challenges the idea that fluent language-model output demonstrates understanding: models learn relationships in human-produced text rather than through direct contact with the world.
Why it matters: It separates a convincing use of language from the question of whether that language is connected to meaning. - Accumulated knowledge versus adaptation: Distinguish crystallized intelligence from fluid intelligence and describe how ARC-AGI tests a model's ability to infer rules and adapt when familiar patterns offer little help.
Why it matters: The distinction explains why strong performance on familiar-looking tasks may not settle whether a system can handle genuinely unfamiliar ones. - Hallucinations and missing grounding: Explain how the stochastic parrot critique connects hallucinations to generating statistically plausible text without reliably checking it against reality, and state the resulting question about what connection to the world understanding requires.
Why it matters: It shows why a fluent answer can be unreliable even when it sounds well formed.
Key terms: stochastic parrot, stochastic, grounding, crystallized intelligence, fluid intelligence, ARC-AGI, core knowledge priors, ARC-AGI-3, hallucinations, grounded representation of reality
9.4.3 A ghost without a shell
This section presents the skeptical case that describing, predicting, or simulating action in the world may not amount to cognition grounded in bodily experience, consequences, agency, and a continuing self.
- Embodied knowledge and world models: Explain why critics distinguish describing the world from learning through bodily interaction with it, and why they question whether language-derived knowledge forms a stable world model.
Why it matters: It clarifies what critics think fluent descriptions may leave out. - Borrowed agency and consequences: Distinguish conversational or tool-supported simulations of agency from the agency critics attribute to beings that pursue goals and face consequences that matter to them.
Why it matters: It separates a system’s ability to carry out plans from the question of whose goals and stakes those plans serve. - Simulated self and lived perspective: Describe why critics distinguish a model’s first-person language or conversational persona from a continuing self, and state the section’s concluding challenge about a lived position from which the world matters.
Why it matters: It identifies the remaining skeptical objection even when a system can describe, predict, and act.
Key terms: bodily skill, practical involvement, world models, stable model of the world, agency, agentic systems, borrowed, external objectives, consequences, self
9.4.4 Different kinds of learning
The section contrasts human and language-model learning to explain why AI can be capable yet static, uneven, and difficult to judge by human expectations.
- Contrasting learning paths: Explain how human learning through relatively few encounters, mistakes, and lasting revisions differs from language models’ data-intensive training before use. Describe the skeptical question this raises without claiming that the models’ abilities are unreal.
Why it matters: How an ability is acquired affects what we can infer from its performance. - Limits of learning during use: Describe why a deployed language model can respond flexibly to prompts and corrections within a conversation yet normally does not retain those corrections or change its underlying model. State the qualifications the section gives for memory systems, fine-tuning, and external learning loops.
Why it matters: A useful correction in one interaction is not necessarily lasting learning. - Uneven abilities and tokenization: Explain why training and representation can give language models an uneven profile: they may handle advanced tasks yet stumble on simple-looking ones. Use the strawberry letter-counting test to illustrate how tokens differ from individual characters, while noting that newer models can solve such tasks with character-level steps or code.
Why it matters: A surprising failure may reveal how a system represents a task, not simply how much it knows.
Key terms: human learning, large-scale training, parameters, general principles, prompts, corrections, underlying model, memory systems, fine-tuning, external learning loops
9.5 The case for machine intelligence
The section introduces a way to assess skeptical arguments about machine intelligence by asking what each alleged missing feature actually rules out.
- Limits of missing features: Distinguish among missing features that expose practical limits, bear on responsibility or personhood, or merely show that AI takes a different route from humans. Explain why each critique must be assessed by what it actually rules out.
Why it matters: A limitation can matter without settling whether a machine is intelligent.
Key terms: reliable reasoning, originality, grounding, agency, selfhood, human-like learning
9.5.1 Intelligence without perfection
The section shows why failures on reasoning benchmarks need careful interpretation and why machine intelligence should be judged by a broad pattern of abilities rather than perfection on any one test.
- Benchmark artifacts and reasoning limits: Distinguish failures caused by a benchmark’s design from evidence of a model’s reasoning limits, using Tower of Hanoi and River Crossing as examples.
Why it matters: A failed puzzle does not, by itself, tell us what ability failed. - Unaided text interface limits: Explain how a static, text-only test can measure unaided execution rather than task-solving with tools, and why a final wrong answer may have several causes.
Why it matters: Separating reasoning from execution and interface limits makes evaluation more informative. - Cumulative case for intelligence: Explain the case for judging general intelligence through an accumulating range of demonstrated abilities, without requiring perfection, universality, human similarity, or superintelligence.
Why it matters: A broad standard takes failures seriously without letting one failure erase many demonstrated capabilities.
Key terms: The Illusion of Thinking, Rethinking the Illusion of Thinking, Tower of Hanoi, River Crossing, long-horizon state tracking, static text interface, unaided textual execution, intermediate states, benchmark limits, general intelligence
9.5.2 Parroting into structure
The section argues that language models’ reliance on prior material and their tendency to hallucinate do not, by themselves, reduce their abstraction, transfer, and novel contributions to mere repetition.
- Novelty beyond mere recombination: Explain why following rules and building on existing material do not rule out novel or cognitively meaningful contributions from language models.
Why it matters: The parrot critique is useful, but it can set a standard for originality that humans could not meet either. - Transfer from token prediction: Describe transfer of learning as reuse of learned structure across tasks, and explain how the mathematics study supports the possibility that next-token prediction develops more than surface copying.
Why it matters: Transfer is the section’s concrete reason for treating some model performance as abstraction rather than isolated imitation. - Hallucination and cognitive absence: Distinguish the reliability problem posed by AI hallucinations from the claim that unreliable output proves cognition is absent, using the comparison with human false memory without equating their mechanisms.
Why it matters: A serious error is reason to check a model’s claims, not by itself an answer to whether the model shows cognition.
Key terms: stochastic parrot critique, Lady Lovelace’s Objection, rule-following, recombination, transfer of learning, next-token prediction, MathCAMPS, internal structures, abstraction, hallucinations
9.5.3 Think outside the body
The section argues that bodily experience matters for intelligence but is not its only possible connection to reality, while acknowledging limits in current AI world models.
- Embodiment as an intelligence test: Explain why lacking human-style bodily experience does not, by itself, rule out intelligence, using the section’s examples of an extraterrestrial communicating through signals and people with different sensory or movement abilities.
Why it matters: It separates the question of intelligence from the question of whether a mind develops as a human mind does. - Knowledge beyond direct experience: Describe how human reasoning and knowledge can depend on symbols, cultural records, instruments, and testimony rather than direct perception or movement, without being detached from reality.
Why it matters: It shows why indirect access to the world can still support meaningful reasoning. - Artificial sensory and action links: Describe how autonomous-driving and robotics systems can gain partial forms of embodiment through sensors, actions, and feedback, without acquiring human biological embodiment.
Why it matters: It prevents treating every AI system as necessarily cut off from the physical world. - Partial language-shaped world models: Explain why language training may produce functional expectations about the physical world without yielding an animal's action-tested world model, and how ARC-AGI-3 illustrates the remaining limitation.
Why it matters: It makes room for useful world knowledge in AI without overlooking its difficulties in unfamiliar environments.
Key terms: embodiment, disembodied intelligence, symbols, representations, inherited cultural systems, partial proxies for embodiment, sensors, actions, feedback, world-model critique
9.5.4 Own will hunting
The section separates intelligence from agency and a human-like self, arguing that their absence changes the kind of AI at issue without settling whether it can think.
- Agency versus intelligence: Distinguish intelligence—understanding, reasoning, planning, and problem solving—from agency, the pursuit of goals as one’s own. Explain why lacking self-originating goals rules out moral autonomy and responsibility but does not, by itself, rule out intelligence.
Why it matters: This distinction prevents a question about responsibility from deciding a different question about thinking. - Functional versus lived self: Describe the limited self-related information current AI can represent, such as its task, tools, limits, instructions, and conversational context. Distinguish this functional self-model from a continuous, lived human self, whose absence does not clearly rule out intelligence.
Why it matters: A useful representation of one’s role and constraints need not amount to a lived identity, and the difference matters when judging AI.
Key terms: agency, intelligence, self-originating goals, moral autonomy, functional self-model, lived autobiography, continuous point of view, memory systems
9.5.5 Different, not worse
The section argues that language models’ different learning paths and uneven abilities warrant caution, but do not by themselves rule out intelligence.
- Human learning head start: Explain why a child’s greater sample efficiency is not a straightforward comparison with a language model’s data needs: newborns have early biases and expectations that help guide learning, while models begin with randomly initialized parameters and acquire their priors through training.
Why it matters: The comparison changes when you account for what each learner starts with. - Frozen learning versus intelligence: Distinguish a system’s ability to keep learning from its ability to use what it has already learned intelligently. Describe the limited anterograde-amnesia analogy the section uses to make that distinction.
Why it matters: A limitation in updating knowledge does not, on its own, settle whether existing abilities are intelligent. - Tool supported flexible competence: Explain why uneven abilities and reliance on tools do not automatically make machine intelligence fake: models can coordinate external resources, just as humans extend thought with aids. State the section’s proposed test—whether a system turns what it has learned into flexible competence, rather than whether it developed exactly as humans do.
Why it matters: It shifts the judgment from resemblance to human development toward what the wider system can do.
Key terms: sample efficiency, priors, randomly initialized parameters, attention biases, frozen after training, continual development, anterograde amnesia, uneven intelligence, external memory, wider system
9.5.6 Thinking, but not like us
This section argues that current AI shows a qualified, unfamiliar form of intelligence: its reasoning and adaptability are difficult to dismiss as mere imitation, but its cognition differs sharply from human thought.
- Qualified machine intelligence: Explain why the section treats current AI as a form of intelligence despite its uneven performance and dependence on training data, prompts, tools, and surrounding systems.
Why it matters: This distinction helps you evaluate AI’s capabilities without treating every failure as proof that they are unreal. - Human and artificial thought: Distinguish the forms of cognition the section attributes to AI from human thought, and explain why its conclusion is neither that AI thinks just like us nor that its behavior is empty simulation.
Why it matters: It gives you a way to discuss whether AI thinks without assuming that human cognition is the only possible model.
Key terms: flexible competence, mere imitation, qualified intelligence, embodied organisms, continuous lived history, original intentionality, grounded perspective, empty simulation
9.6 Summary
The chapter summary weighs the evidence for machine intelligence against critiques of imitation, reasoning, grounding, and embodiment, reaching a qualified judgment about current AI.
Concepts explained in this chapter
- intelligenceThe capacity to perform cognitive activities such as learning, reasoning, and problem solving, as inferred here from observable behavior.
- emergent behaviorHigher-level system behavior arising from interactions among components without being explicitly programmed.
- reasoning modelA reasoning model is a language model designed and trained to perform intermediate reasoning computation before producing an answer.
- hallucinationAn AI hallucination is a generated claim or content that is false, unsupported, or fabricated while being presented as plausible.
- stochastic parrot critiqueThe stochastic parrot critique is the argument that language models generate plausible language by recombining learned patterns without genuine understanding or intent.
- world modelA world model is an internal computational model that represents entities, their relations, and how situations evolve so a system can reason about or act on a world.
- trainingLearning is the acquisition or improvement of knowledge, representations, or capabilities through experience or data.
- artificial intelligenceThe field of creating computational systems that perform tasks associated with human intelligence.
- thinkingThe mental process of forming thoughts, reasoning, or engaging in conscious cognition.
- language modelA computational model of patterns in language that can assign probabilities to or generate language sequences.
- lived experienceFirst-person experience of perceiving, acting in, and encountering the world.
- embodimentEmbodiment is the dependence of cognition or understanding on a physical body and its situated interaction with the world.
- agencyAgency is the capacity to initiate actions and pursue goals as an agent.
- disembodied intelligenceDisembodied skill is competent performance detached from physical situation, lived experience, or direct worldly engagement.
- intentionalityIntentionality is the capacity of a mental state to be about or directed toward something.
- simulationSimulation is the computational or formal reproduction of the behavior or operation of a system.
- chain-of-thoughtReasoning that proceeds through an explicit sequence of intermediate inferential steps.
- adaptive imitationAdaptive imitation is behavior that preserves an appearance of capability by adjusting successfully when conditions change.
- brittle imitationBrittle imitation is behavior that appears capable under narrow conditions but fails when those conditions change.
- imitationImitation intelligence is the appearance of intelligence produced by convincingly reproducing intelligent behavior without genuine understanding.
- reasoningThe process of deriving conclusions or solutions from information, premises, or relationships.
- goalA desired outcome or state that guides an agent's behavior or decisions.
- next-word predictionNext-token prediction is the task of predicting the token that follows a given sequence of preceding tokens.
- transfer learningTransfer of learning is the reuse of knowledge or structure learned in one context to support performance in another context.
- data representationA structured encoding of information that can support reasoning or computation.
- fine-tuningFine-tuning is the continued adjustment of a pretrained model on narrower data or objectives for a particular task or behavior.
- tokenA discrete unit of text used in computational language processing, such as a character, word fragment, or word.
- evaluationEvaluation is the systematic assessment of a model's outputs or capabilities against specified criteria or reference results.
- human intelligenceThe human capacity for language, reasoning, adaptation, creativity, and related cognitive abilities.
- performanceThe observable execution or result of a task, as distinct from the capacity underlying it.
- chain-of-thought promptingA prompting technique that encourages a model to produce intermediate reasoning steps before its final answer.
- groundingGrounding is the connection of symbols or representations to the real-world entities, situations, and experiences they refer to.
- cognitionMental processes involved in acquiring, processing, storing, and applying information.
- understandingUnderstanding is comprehension of the meaning, structure, or significance of information rather than merely producing an appropriate response.
- neural networkA computational model composed of interconnected units with parameters that can be learned to process patterns.
- neuronA computational unit in a neural network that transforms incoming signals and passes an output to other units.
- memoryThe capacity to encode, retain, and retrieve information, including information carried between successive sequence positions.
- supervised fine-tuningSupervised fine-tuning adjusts a pretrained model using labeled examples of desired inputs and outputs or behaviors.
- computeProcessing capacity available to run computational workloads.
- contextContext is the surrounding information and circumstances that determine how an expression, event, or task should be interpreted.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
