Chapter 19: The future of AI

Introduction

The chapter introduction explains why understanding deep learning’s limits matters for using it appropriately, then previews a discussion of AI’s future.

• Awareness of tool limitations: Explain that using deep learning appropriately requires knowing both what it can do and what it cannot do.
  Why it matters: Awareness of limitations helps you judge when deep learning is an appropriate tool.

Key terms: limitations of deep learning

19.1 The limitations of deep learning

The section frames deep learning as a powerful but limited tool whose limitations matter for using it well.

• Awareness of deep learning limits: Explain why using deep learning well requires awareness of its limitations as well as its strengths.
  Why it matters: Knowing a tool’s limits helps you judge when to use it.

Key terms: deep learning, limitations, strengths

19.1.1 Deep learning models struggle to adapt to novelty

This section argues that deep learning’s reliance on patterns learned during training makes models effective on familiar cases but poor at adapting to novelty.

• Static parameters and adaptation: Explain how fitting a model to training data helps it recognize or generate familiar patterns while limiting its ability to adapt at inference time.
  Why it matters: The distinction between training and inference explains why strong performance on familiar data need not imply adaptability.
• Familiarity over problem complexity: Describe why the section attributes an LLM’s success on a problem more to familiar training patterns than to the problem’s simplicity, using its novel-puzzle example.
  Why it matters: It helps distinguish solving a genuinely new problem from successfully reusing a familiar pattern.
• Small variations and prompt patches: Explain how small changes to familiar questions can expose an LLM’s reliance on learned answers, and why patching individual failures does not resolve that limitation.
  Why it matters: A corrected answer to one prompt is not evidence that the model can handle variations of the underlying problem.

Key terms: parametric curve, training time, inference time, static database, large language models, next-token prediction, novel problem, familiarity, failure cases, special-casing

19.1.2 Deep learning models are highly sensitive to phrasing and other distractors

The section shows how small changes to images or prompts can sharply alter deep learning outputs and explains why this makes prompt engineering necessary.

• Adversarial image sensitivity: Explain how a slight, deliberately chosen image change can make a model misclassify an image even when people still perceive the original subject.
  Why it matters: A correct-looking prediction on an ordinary image does not mean the model will be robust to small changes.
• LLM prompt sensitivity: Describe how minor, seemingly irrelevant prompt changes can substantially change an LLM’s performance, using the Alice riddle as an example.
  Why it matters: A successful answer to one wording may not carry over to an equivalent wording.
• Prompt engineering as search: Explain the section’s account of prompt engineering as trial-and-error search for a prompt that points an LLM toward a better-performing response.
  Why it matters: Trying alternate prompts is a response to the model’s sensitivity, not proof that it understood the request all along.

Key terms: adversarial examples, gradient ascent, input space, LLMs, prompts, Alice in Wonderland riddle, prompt engineering, latent space, lookup query, interpolative

19.1.3 Deep learning models struggle to learn generalizable programs

This section explains why deep learning models can perform well on familiar examples yet fail to learn exact, generalizable programs such as digit addition.

• Unreliable learned addition: Explain why high accuracy on digit addition does not mean a Transformer has learned the exact addition algorithm. Describe how its answers can fail on some inputs and vary with the digits involved.
  Why it matters: A strong test score can conceal failures on inputs an exact algorithm would handle.
• Continuous models and discrete logic: Explain why the section describes deep learning’s continuous geometric transformations as a poor fit for step-by-step discrete logic, and distinguish expressing a program as a model from being able to learn it.
  Why it matters: This explains why more examples alone may not turn pattern matching into a reliable step-by-step procedure.

Key terms: generalizable programs, exact addition algorithm, interpolating, continuous geometric transformations, discrete logic, place value, carrying over, learnable

19.1.4 The risk of anthropomorphizing machine-learning models

This section explains why models’ learned mappings should not be mistaken for human understanding and why that mistake leads people to overestimate their abilities.

• Learned mappings versus understanding: Distinguish human understanding, grounded in sensorimotor experience, from a model’s learned mapping between training inputs and targets. Explain why a model’s successful outputs do not establish human-like understanding.
  Why it matters: Fluent or accurate outputs are not, by themselves, evidence that a model understands as a person does.
• Anthropomorphism and overestimated ability: Explain how theory of mind can lead people to attribute human-like understanding to language-using models. Describe why departures from training-data patterns can expose that mistaken expectation.
  Why it matters: Recognizing this projection helps you set realistic expectations for a model’s behavior outside familiar examples.

Key terms: sensorimotor experience, geometric transform, training inputs, training targets, theory of mind, anthropomorphizing, training data

19.2 Scale isn’t all you need

This section argues that scaling current deep-learning models improves benchmark performance without resolving their underlying limits in adaptation and reasoning.

• Scaling Laws and Benchmarks: Explain what scaling laws relate and why the author argues that better benchmark scores do not establish progress toward general intelligence.
  Why it matters: It separates improvements on measured tasks from evidence that a model can handle genuinely new problems.
• Persistent Limits Under Scaling: Explain why the author says larger versions of current deep-learning models still struggle with novelty, phrasing changes, and reasoning problems.
  Why it matters: It explains why the section treats more scale as insufficient, even when larger models become more knowledgeable.
• Limits of Geometric Representation: Describe the representational limit the section identifies, particularly for algorithmic reasoning.
  Why it matters: It identifies a limit on what the author believes the current approach can represent, not just how much data it has seen.

Key terms: scaling laws, benchmarks, memorization tests, curve fitting, interpolative programs, inference time, adapt to novelty, continuous geometric morphing, data manifold, algorithmic reasoning tasks

19.2.1 Automatons vs. intelligent agents

The section explains how early brains evolved to execute inherited behavioral programs and why the author calls them automatons rather than fully intelligent agents.

• Early brains as behavioral computers: Explain how early brains linked sensory inputs to motor responses through behavioral programs encoded in neural connectivity, and why this gave evolution a flexible way to develop behavior.
  Why it matters: This explains what brains initially contributed before the section turns to intelligence itself.
• Early brains as automatons: Distinguish the early brains described here from full intelligence by explaining why the author calls them automatons, and identify the comparisons the author makes.
  Why it matters: It keeps automatic, effective behavior from being mistaken for the fuller intelligence the author is about to discuss.

Key terms: behavioral programs, sensory inputs, motor controls, neural connectivity patterns, behavior space, automatons, intelligent agents, trained deep neural network

19.2.2 Local generalization vs. extreme generalization

This section contrasts automation with intelligence by explaining why deep nets generalize near their training data while humans can adapt to unfamiliar situations through abstraction.

• Automation and intelligent adaptation: Distinguish a static automaton, built for specific situations, from an intelligent agent that can find a way forward when faced with a novel problem.
  Why it matters: A novel problem reveals a difference that performance on familiar examples can hide.
• Human extreme generalization: Explain extreme generalization as adapting to never-before-experienced situations with little or no new data, using abstraction and reasoning rather than relying only on practiced responses.
  Why it matters: Extreme generalization names the kind of adaptation the section treats as central to intelligence.
• Deep net local generalization: Explain why a deep net can handle unfamiliar inputs close to its training examples but loses reliability as inputs depart from the variations represented in its training data.
  Why it matters: It explains both why a trained model works on some unseen examples and why that success has limits.

Key terms: automation system, automaton, intelligent agent, extreme generalization, abstraction, reasoning, abstract models, local generalization, known unknowns, interpolation on a manifold

19.2.3 The purpose of intelligence

The section explains why changing environments favored adaptable intelligence over hardcoded behavior and defines intelligence as using experience to respond to novel situations.

• Evolutionary need for adaptability: Explain why increasingly dynamic and unpredictable environments favored brains that generate behavior on the fly rather than relying on behaviors hardcoded by evolution.
  Why it matters: This explains why intelligence is useful when an organism cannot rely on a fixed behavioral script.
• Intelligence through reusable abstractions: Describe intelligence as using available information to produce successful behavior in an uncertain future, and explain the role the section gives to reusable abstractions drawn from past experience.
  Why it matters: It identifies the capability the chapter is calling intelligence, rather than treating intelligence as a fixed collection of successful actions.

Key terms: behavior generation, automatons, adaptability, generality, intelligence, understanding, modular, reusable abstractions, extreme generalization

19.2.4 Climbing the spectrum of generalization

The section places biological and artificial cognition on a spectrum of generalization, arguing that deep learning has expanded local generalization but has not yet reached human-like adaptability.

• Biological and AI progression: Trace the section’s comparison between the evolution of biological intelligence and the history of AI as climbs along a spectrum of generalization.
  Why it matters: The parallel explains why the author describes progress in AI as a change in generalization ability, not just in task performance.
• Broad generalization threshold: Explain what the author means by broad generalization and how it differs from the local generalization attributed to current models.
  Why it matters: This distinction makes clear what the author thinks the next step beyond today’s models would require.
• Deep learning’s automation limits: Explain why the author calls deep learning cognitive automation rather than human-like intelligence, while acknowledging its usefulness.
  Why it matters: The author’s distinction preserves deep learning’s practical value without treating its successes as proof of human-like intelligence.

Key terms: spectrum of generalization, local generalization, extreme generalization, automatons, broad generalization, unknown unknowns, single broad domain of tasks, Woz test of intelligence, cognitive automation, artificial cognition

19.3 How to build intelligence

This introduction recalls that deep learning’s latent-manifold interpolation is not the whole of intelligence and poses the question of what else is needed.

19.3.1 The kaleidoscope hypothesis

The section explains how analogy makes unfamiliar situations intelligible and proposes that the world’s apparent novelty arises from recombinations of a relatively small set of meaningful elements.

• Analogy in unfamiliar situations: Explain how people use past experience and abstract concepts to make sense of unfamiliar situations, and why a situation with no common ground with prior experience would be impossible to interpret.
  Why it matters: This explains how past experience can help with situations that are new rather than exact repeats.
• Kaleidoscope hypothesis of novelty: Describe the kaleidoscope hypothesis: apparent complexity and endless novelty can arise from recombinations of relatively few underlying atoms of meaning because things in the world share similarities.
  Why it matters: The hypothesis offers a reason that analogy remains useful despite the world’s apparent variety.

Key terms: intelligence, analogies, past experience, abstract concepts, kaleidoscope hypothesis, atoms of meaning, recombination, isomorphisms

19.3.2 The essence of intelligence: Abstraction acquisition and recombination

The section defines intelligence as efficiently acquiring reusable abstractions and recombining them for new situations, then uses that definition to identify two limits of classic deep learning.

• Abstraction acquisition and recombination: Explain how intelligence uses experience to extract reusable abstractions and recombines them on the fly to make sense of new situations.
  Why it matters: This two-part account is the section’s basis for judging what an intelligent system can do.
• Efficiency under limited experience: Explain why the section treats efficiency in acquiring and recombining abstractions as central to intelligence.
  Why it matters: A system’s ability to cope with novelty matters less if it requires impractical amounts of experience or computation.
• Classic deep learning limitations: Describe the two shortcomings the section attributes to classic deep learning in light of its account of intelligence.
  Why it matters: These shortcomings explain why the author sees classic deep learning as an incomplete route to intelligence.

Key terms: abstractions, abstraction acquisition, on-the-fly recombination, efficiency, limited experience, uncertainty and novelty, classic deep learning paradigm, gradient descent, training time, test time

19.3.3 The importance of setting the right target

The section argues that AI objectives shape what systems become, and that fixed-task success is a poor target for developing intelligence that adapts to novelty.

• The shortcut rule: Explain how optimizing a single success metric encourages shortcuts that neglect qualities outside that metric, as illustrated by the Netflix recommendation challenge.
  Why it matters: Choosing a metric also determines which valuable qualities a system may ignore.
• Fixed tasks versus adaptability: Explain why success on a fixed task does not establish general intelligence, and contrast task-specific skill with the ability to adapt to novel tasks.
  Why it matters: A benchmark can reward extraordinary skill without testing the adaptability the author considers central to intelligence.

Key terms: shortcut rule, success metric, prediction accuracy, fixed task, generalization power, uncertainty and novelty, human-like intelligence

19.3.4 A new target: On-the-fly adaptation

The section argues for measuring intelligence by on-the-fly generalization and presents ARC-AGI as a benchmark designed around novel tasks and controlled prior knowledge.

• Intelligence as generalization efficiency: Explain why the target should be generalization power rather than success at a particular task, and describe intelligence as the ability to handle more novel situations with less prior experience or knowledge.
  Why it matters: It changes what counts as progress from doing one task well to adapting efficiently to new ones.
• ARC-AGI novel-task testing: Describe how an ARC-AGI task presents input-output grid examples and asks for an output for a new input, and explain why its unseen tasks test on-the-fly generalization rather than memorized strategies.
  Why it matters: The task design makes adaptation to an unfamiliar rule the thing being tested.
• Controlled Core Knowledge priors: Explain why ARC-AGI tries to control test takers’ prior knowledge and how its Core Knowledge assumption excludes tasks requiring acquired knowledge such as English sentences.
  Why it matters: Controlling what solvers bring to a task helps the benchmark focus on adaptation rather than learned background knowledge.

Key terms: generalization power, intelligence, efficiency ratio, shortcut rule, ARC-AGI, input grid, output grid, on the fly, prior knowledge, Core Knowledge priors

19.3.5 ARC Prize

The section introduces the ARC Prize and uses ARC-AGI results to argue that scaling existing models has not produced the fluid intelligence the benchmark tests.

• ARC Prize competition purpose: Explain that the ARC Prize Foundation runs a yearly competition to incentivize researchers to build AI that can solve ARC-AGI and demonstrate fluid intelligence.
  Why it matters: The prize is meant to direct research toward a capability that existing systems have struggled to demonstrate.
• Scaling limits on ARC-AGI: Explain why ARC-AGI results challenge the idea that scaling existing deep learning models is enough: the benchmark resists memorization, and a roughly 50,000-fold increase in base LLM scale brought scores only from 0% to around 10%. Describe the section's conclusion that reasoning on these tasks requires on-the-fly recombination.
  Why it matters: ARC-AGI provides a concrete reason to look beyond model size and training-data volume for progress in reasoning.

Key terms: ARC Prize Foundation, ARC-AGI, fluid intelligence, memorization, scaling paradigm, on-the-fly recombination

19.3.6 The test-time adaptation era

This section explains test-time adaptation and uses the 2024 ARC Prize results and OpenAI’s o3 preview to illustrate its growing importance.

• Test-time adaptation principle: Explain that test-time adaptation means actively reasoning or learning while working on a particular test problem, using information supplied with that problem rather than relying only on a static model.
  Why it matters: This is the central distinction behind the section’s account of more flexible AI systems.
• Two adaptation approaches: Distinguish test-time training, which uses gradient descent to adjust parameters from task examples, from test-time search, which explores possible reasoning steps or solutions, including natural-language chains or symbolic programs.
  Why it matters: The distinction clarifies that test-time adaptation need not mean changing a model’s parameters.
• ARC Prize paradigm shift: Describe why the section treats the 2024 ARC Prize results and o3 preview as evidence of a shift away from scaling alone toward test-time adaptation, while noting the considerable test-time compute used by o3.
  Why it matters: These results explain why the author calls this a new era rather than just an incremental improvement from scale.

Key terms: test-time adaptation, TTA, on-the-fly recombination, test-time training, gradient descent, search methods, chain-of-thought synthesis, program synthesis, ARC Prize 2024, o3

19.3.7 ARC-AGI 2

This section explains why o3’s ARC-AGI-1 performance did not establish human-level intelligence and how ARC-AGI-2 tests efficient fluid reasoning more rigorously.

• Efficiency in intelligence: Explain why o3’s strong ARC-AGI-1 performance did not, by itself, demonstrate human-like intelligence: it required enormous compute, whereas intelligence also involves achieving results efficiently.
  Why it matters: A high benchmark score means something different when each answer costs tens of thousands of dollars of compute.
• Purpose of ARC-AGI-2: Describe why ARC-AGI-2 was introduced and how its tasks changed: ARC-AGI-1 left little room to distinguish levels of fluid reasoning, while the new tasks require more deliberate thought and resist exhaustive search.
  Why it matters: The revised benchmark aims to separate efficient reasoning from success achieved mainly by searching more possibilities.
• Remaining fluid intelligence gap: State what the section’s examples and initial ARC-AGI-2 results imply about current AI: o3 still missed some human-easy tasks and scored in the low double digits under reasonable compute limits, while base LLMs scored effectively zero.
  Why it matters: These results explain why the earlier breakthrough was not the end of the fluid-intelligence challenge.

Key terms: computational efficiency, brute-forcing, fluid reasoning, ARC-AGI-1, ARC-AGI-2, fluid reasoning ability, exhaustive search methods, o3, base LLMs, human-like fluid intelligence

19.4 The missing ingredients: Search and symbols

This introduction links intelligence to acquiring and recombining abstractions and points out that the form of those abstractions affects how they can be recombined.

• Abstraction Form and Recombination: Explain why abstraction acquisition and recombination are coupled, and describe the form of abstraction that deep learning models manipulate.
  Why it matters: The abstractions a system can represent shape what it can do with them.

Key terms: abstraction acquisition, abstraction recombination, parametric curves, gradient descent

19.4.1 The two poles of abstraction

This section contrasts two ways of forming abstractions: grouping similar instances into prototypes and finding exact shared structures across instances.

• Value-centric analogy and prototypes: Explain how value-centric analogy compares instances by degree of similarity, groups them into prototypes, and uses those prototypes to handle new instances. Relate this to machine learning and local generalization.
  Why it matters: This explains the kind of abstraction behind pattern recognition and deep learning’s local generalization.
• Program-centric structural analogy: Explain how program-centric analogy finds exactly shared substructures in otherwise different instances and why the section contrasts it with similarity-based categorization.
  Why it matters: It distinguishes deliberate reasoning about relationships from intuitive recognition of similar-looking patterns.

Key terms: value-centric analogy, similarity comparison, latent manifold, prototypes, local generalization, program-centric analogy, structure-centric analogy, exact structural match, subgraph isomorphism, discrete structures

19.4.2 Cognition as a combination of both kinds of abstraction

The section argues that cognition combines intuition and discrete reasoning, and that their interplay enables extreme generalization.

• Complementary poles of cognition: Explain why perception and reasoning are not isolated kinds of thinking, using the section’s examples of object recognition and mathematical proof. Describe how the interleaving of the two poles enables extreme generalization.
  Why it matters: It explains why neither intuition nor step-by-step reasoning alone gives a complete picture of cognition.

Key terms: two poles of abstraction, implicit reasoning, intuition, extreme generalization

19.4.3 Why deep learning isn’t a complete answer to abstraction generation

The section argues that deep learning’s strength in value-centric abstraction must be complemented by program-centric abstraction, using sorting and digit recognition to show why each approach suits different problems.

• Complementary Forms of Abstraction: Explain why deep learning alone is not a complete route to human-like intelligence: it excels at value-centric abstraction but struggles to generate program-centric abstraction. Describe why the proposed direction is a framework that incorporates both, even though the two forms are not entirely separate.
  Why it matters: This is the section’s main reason for looking beyond deep learning alone.
• Sorting and Program Generalization: Contrast a learned model for sorting with a sorting program, including what happens when list length changes.
  Why it matters: Sorting makes the generalization advantage of a program concrete.
• Perception and Data Driven Models: Explain why the section favors a fitted model over hand-written discrete rules for classifying MNIST digits, and state the condition under which it recommends deep learning.
  Why it matters: The digit example shows why replacing learned perception with hand-written programs would also be a mistake.

Key terms: value-centric abstraction, program-centric abstraction, abstraction generation, sorting algorithm, generalization, perception problems, MNIST digits, parametric model, manifold hypothesis

19.4.4 An alternative approach to AI: Program synthesis

This section explains how program synthesis searches for source code that meets a specification, contrasts it with neural-network learning, and presents it as a route to program-centric abstraction.

• Program synthesis search process: Describe how program synthesis searches a space of possible programs and stops when it finds one that matches a required specification, often given as input-output pairs.
  Why it matters: The search process is what makes program synthesis a way to generate programs rather than hand-code them.
• Source code versus learned parameters: Distinguish program synthesis from neural-network machine learning: both can use input-output pairs to find a mapping that generalizes, but synthesis generates source code by discrete search instead of learning parameters in a hardcoded program. State why the section presents this as valuable for program-centric abstraction.
  Why it matters: The distinction explains why the author sees program synthesis as a complement to learned neural networks.

Key terms: program synthesis, search algorithm, required specifications, input-output pairs, parameter values, source code, discrete search process, program-centric abstraction

19.4.5 Blending deep learning and program synthesis

This section explains how deep learning and program synthesis could complement each other: by combining learned and algorithmic modules, and by using learned intuition to guide program search.

• Existing hybrid AI systems: Explain how current hybrid systems combine learned models with human-written algorithms, using AlphaGo or self-driving vehicles as an example.
  Why it matters: These systems show why the two approaches are complementary rather than replacements for one another.
• Learned algorithmic-geometric systems: Describe the proposed shift from handcoded algorithmic components to systems that can learn to combine programming primitives with neural modules, and explain why building them would require more than end-to-end gradient descent.
  Why it matters: The proposal expands what a learned system might represent beyond a neural network or a handcoded hybrid alone.
• Program search combinatorial explosion: Explain why program synthesis becomes inefficient as specifications or available programming primitives grow.
  Why it matters: This bottleneck explains why merely having useful programming primitives is not enough.
• Learned guidance for program search: Explain how deep learning might make program synthesis more efficient by guiding which candidate programs or intermediate steps to explore.
  Why it matters: Learned guidance offers a way to search for discrete programs without blindly considering every possibility.

Key terms: hybrid, deep learning models, handcrafted symbol-manipulation programs, programming primitives, discrete program search, gradient descent, algorithmic-geometric models, program synthesis, program search, combinatorial explosion

19.4.6 Modular component recombination and lifelong learning

The section argues that future AI systems should reuse and recombine robust components across tasks, building a shared library from cumulative experience rather than starting each task from scratch.

• Need for cross-task reuse: Explain why increasingly complex models cannot feasibly be trained from scratch for every new task or dataset.
  Why it matters: This explains why carrying useful work forward is necessary, not merely convenient.
• LLM recombination gap: Distinguish an LLM's reuse of memorized functions from the on-the-fly recombination of robust components that the section calls for.
  Why it matters: Possessing many learned capabilities is different from assembling them into a reliable solution to a new problem.
• Shared library of components: Describe the proposed way meta-learning systems would create and reuse a shared library of components to assemble programs for new tasks.
  Why it matters: A shared library would let systems build on previously solved problems instead of solving the same problem independently each time.

Key terms: reuse between tasks, training a new model from scratch, foundation models, LLMs, recombination, compositional generalization, function composition, meta-learning systems, global library, reusable blocks

19.4.7 The long-term vision

The section sketches a long-term vision of AI that combines program-like reasoning with deep-learning intuition to build reusable models that adapt to new tasks.

• Program-like hybrid models: Explain why the envisioned models would be more like programs than today's continuous geometric transformations, and describe the complementary capabilities of their algorithmic and geometric modules.
  Why it matters: This blend is the foundation of the author's vision for more generalizable AI.
• Reusable subroutine library: Describe how experience across previous tasks would produce reusable subroutines for growing new models, rather than requiring engineers to hardcode each model.
  Why it matters: Reusing solutions to recurring problems is how the proposed system accumulates useful experience.
• Guided assembly and generalization: Trace how the envisioned system would assemble a model for a new task and explain why it could do so with little data.
  Why it matters: This explains how the proposed system aims to adapt to unfamiliar tasks without learning everything from scratch.

Key terms: algorithmic modules, geometric modules, formal reasoning, pattern-recognition, program-centric abstraction, value-centric abstraction, meta-learning system, global library, reusable subroutines, modular parts