liveBook · Manning

Chapter 9: When machines seem to think

What you will learn in chapter 9 of Machines that Think

The chapter introduces a balanced way to judge whether current AI can count as intelligent without treating either its failures or its impressive performance as decisive.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces a balanced way to judge whether current AI can count as intelligent without treating either its failures or its impressive performance as decisive.

Key terms: hallucinations, bias, brittleness, grounding, convincing performance, observable performance, competence, surface imitation, mechanism, agency

9.1 When it walks like a duck

This section introduces the tension between judging an AI system by its convincing behavior and judging it by the mechanism that produces that behavior.

Key terms: performance, statistical learning, perception, intention, experience

9.1.1 A convincing performance

This section contrasts brittle and adaptive imitation to explain why language models’ convincing performance raises, but does not settle, the question of machine intelligence.

Key terms: brittle imitation, adaptive imitation, ELIZA, chess engine, large language models, intelligence, lived experience, nonbiological mechanisms, Turing

9.1.2 A tricky definition

The section explains why intelligence must be inferred from performance and why language models’ observable abilities make their status difficult to settle.

Key terms: intelligence, thought, patterns of performance, construct, psychometrics, observable performance, context, learned patterns, internal parameters, problem solving

9.1.3 Intelligence in others

The section argues that intelligence is judged relative to a system’s expected capacities, and that this matters when evaluating animal cognition and language models.

Key terms: expected baseline, comparative cognition, species-specific capacities, mirror self-recognition, visible mark, bodily self-recognition, language models, flexibility, bias of familiarity, imitation

9.1.4 What performance is worth

This section weighs what flexible performance can show about machine intelligence while distinguishing outward success from competence, instantiation, and intentionality.

Key terms: performance, competence, master copyist, simulation, instantiation, functionalism, intentionality, original intentionality, derived intentionality

9.2 Thought behind chains

This introduction explains why step-by-step generation makes the question of whether language models think harder to answer.

Key terms: chain-of-thought, reasoning models, token, generation

9.2.1 Thinking token after token

This section explains how generating intermediate steps can improve a language model’s answers and how several methods extend that approach.

Key terms: chain-of-thought, intermediate reasoning steps, context, token by token, self-consistency, Tree of Thoughts, Graph of Thoughts, STaR, Self-Taught Reasoner, rationales

9.2.2 A new member of the family

This section explains how reasoning models turn prompted step-by-step reasoning into trained, sometimes hidden computation, while cautioning that improved performance does not prove machine thought.

Key terms: reasoning models, chain of thought, inference-time computation, intermediate work, internal, hidden, compressed, final response, supervised fine-tuning, reward-based training

9.2.3 Lessons from reasoning

The section explains how giving a model room for intermediate steps can reveal reasoning-like performance, while cautioning that those steps do not necessarily reveal how the answer was produced.

Key terms: chain-of-thought, intermediate structure, context, partial results, explanation, computation, scaffold, misleading trace, reasoning-like behavior, reasoning mechanisms

9.3 More than the sum

The section introduces the question of how simple learned operations can, through large-scale interaction, produce complex behavior.

Key terms: artificial neural networks, learned relationships, at scale

9.3.1 When ability appears

The section explains how abilities can become visible in larger language models, while cautioning that the appearance of a sudden new ability can depend on how performance is measured.

Key terms: emergent property, complex systems science, local interactions, emergent abilities, scale, apparent discontinuity, token generation, metrics, benchmark, mirage

9.3.2 The recipe for emergence

This section explains how interacting training conditions can produce apparently sudden abilities, distinguishes that from grokking, and shows how usage conditions can reveal latent capacities.

Key terms: pattern accumulation, pattern interaction, intermediate representations, coordination across components, threshold dynamics, grokking, training loss, validation loss, generalization, amplifiers

9.3.3 From emergence to intelligence

The section argues that emergent abilities make token prediction an incomplete description of a language model’s competence, while leaving open whether that competence amounts to intelligence.

Key terms: token generation, learned relationships, emergent behavior, competence, emergence, imitation, reliable, flexible, transferable, biological cognition

9.4 The illusion hypothesis

The section reframes skepticism about machine intelligence: token-by-token prediction alone does not rule it out, so skeptics must identify an essential element that mechanism cannot provide.

Key terms: emergence, token-by-token prediction, essential to intelligence

9.4.1 The illusion of thinking

This section uses controlled puzzle tests to examine where reasoning models succeed, where they fail, and what those failures do—and do not—show about machine reasoning.

Key terms: GPT-4 Can’t Reason, Large Language Models Are Not Strong Abstract Reasoners, GSM-Symbolic, The Illusion of Thinking, Tower of Hanoi, Checker Jumping, River Crossing, Blocks World, initial state, three regimes

9.4.2 It’s just a parrot

This section explains the stochastic parrot critique of language models, using unfamiliar-task benchmarks and hallucinations to question whether fluent language is grounded in understanding.

Key terms: stochastic parrot, stochastic, grounding, crystallized intelligence, fluid intelligence, ARC-AGI, core knowledge priors, ARC-AGI-3, hallucinations, grounded representation of reality

9.4.3 A ghost without a shell

This section presents the skeptical case that describing, predicting, or simulating action in the world may not amount to cognition grounded in bodily experience, consequences, agency, and a continuing self.

Key terms: bodily skill, practical involvement, world models, stable model of the world, agency, agentic systems, borrowed, external objectives, consequences, self

9.4.4 Different kinds of learning

The section contrasts human and language-model learning to explain why AI can be capable yet static, uneven, and difficult to judge by human expectations.

Key terms: human learning, large-scale training, parameters, general principles, prompts, corrections, underlying model, memory systems, fine-tuning, external learning loops

9.5 The case for machine intelligence

The section introduces a way to assess skeptical arguments about machine intelligence by asking what each alleged missing feature actually rules out.

Key terms: reliable reasoning, originality, grounding, agency, selfhood, human-like learning

9.5.1 Intelligence without perfection

The section shows why failures on reasoning benchmarks need careful interpretation and why machine intelligence should be judged by a broad pattern of abilities rather than perfection on any one test.

Key terms: The Illusion of Thinking, Rethinking the Illusion of Thinking, Tower of Hanoi, River Crossing, long-horizon state tracking, static text interface, unaided textual execution, intermediate states, benchmark limits, general intelligence

9.5.2 Parroting into structure

The section argues that language models’ reliance on prior material and their tendency to hallucinate do not, by themselves, reduce their abstraction, transfer, and novel contributions to mere repetition.

Key terms: stochastic parrot critique, Lady Lovelace’s Objection, rule-following, recombination, transfer of learning, next-token prediction, MathCAMPS, internal structures, abstraction, hallucinations

9.5.3 Think outside the body

The section argues that bodily experience matters for intelligence but is not its only possible connection to reality, while acknowledging limits in current AI world models.

Key terms: embodiment, disembodied intelligence, symbols, representations, inherited cultural systems, partial proxies for embodiment, sensors, actions, feedback, world-model critique

9.5.4 Own will hunting

The section separates intelligence from agency and a human-like self, arguing that their absence changes the kind of AI at issue without settling whether it can think.

Key terms: agency, intelligence, self-originating goals, moral autonomy, functional self-model, lived autobiography, continuous point of view, memory systems

9.5.5 Different, not worse

The section argues that language models’ different learning paths and uneven abilities warrant caution, but do not by themselves rule out intelligence.

Key terms: sample efficiency, priors, randomly initialized parameters, attention biases, frozen after training, continual development, anterograde amnesia, uneven intelligence, external memory, wider system

9.5.6 Thinking, but not like us

This section argues that current AI shows a qualified, unfamiliar form of intelligence: its reasoning and adaptability are difficult to dismiss as mere imitation, but its cognition differs sharply from human thought.

Key terms: flexible competence, mere imitation, qualified intelligence, embodied organisms, continuous lived history, original intentionality, grounded perspective, empty simulation

9.6 Summary

The chapter summary weighs the evidence for machine intelligence against critiques of imitation, reasoning, grounding, and embodiment, reaching a qualified judgment about current AI.

Concepts explained in this chapter

← Chapter 8: The impact of AI

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.