liveBook · Manning

Chapter 13: Expanding the boundaries of learning: OpenAI and the scaling laws of neural language models

What you will learn in chapter 13 of Timeless Algorithms, The Foundational Ideas

This introduction connects earlier learning ideas and transformer architecture to evidence that scaling model parameters, data, and compute yields predictable gains while leaving important constraints in place.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction connects earlier learning ideas and transformer architecture to evidence that scaling model parameters, data, and compute yields predictable gains while leaving important constraints in place.

Key terms: scaling laws, language-model loss, power-law relationships, parameters, training data, computational budgets, sample efficiency, generalization, transfer across tasks, constraints on learning

13.1 Discovering the scaling relationships

The section explains why better performance from larger language models raised an empirical question and why answering it required controlled comparisons across models.

Key terms: capacity, training data, compute, model size, general empirical relationship

13.1.1 The empirical puzzle

This section frames the empirical puzzle of why larger language models kept improving on unseen text and explains how large-scale comparisons made that improvement a measurable research question.

Key terms: generalization, parameters, cross-entropy loss, held-out text, performance plateau, GPT, model capacity, training data, compute, distributed computing infrastructure

13.1.2 Measuring performance across resources

This section explains how researchers compared transformer language models across model size, training data, and compute using a consistent prediction task and held-out loss.

Key terms: transformer-based language models, next-token prediction objective, model size, parameters, training-data volume, tokens, compute, cross-entropy loss, held-out text, next-token prediction

13.1.3 The empirical pattern

This section identifies a consistent empirical pattern in language-model scaling, shows how it supports performance estimates, and explains why those estimates are conditional.

Key terms: held-out cross-entropy loss, model size, training-data volume, compute, binding constraint, power laws, logarithmic axes, slope, orders of magnitude, scaling laws

13.2 Parameters, data, compute, and power laws

The section explains why scaling language models requires coordinating parameters, training data, and compute rather than increasing any one resource alone.

Key terms: parameters, training data, compute, statistical complexity, scaling, binding constraint, diminishing value, power laws

13.2.1 Model parameters: representational capacity

This subsection explains how parameter count expands a language model’s representational capacity and contrasts the expected risk of overfitting with an observed scaling pattern.

Key terms: parameters, weights and biases, representational capacity, deeper, wider, overfitting, held-out loss, power-law relationship, training data, compute

13.2.2 Training data: available information

This section explains how training data supplies evidence for language models, how its volume relates to prediction performance, and why more tokens are not always more useful.

Key terms: parameters, training data, tokens, sequences, model capacity, held-out loss, power law, training tokens, next-token prediction, compute

13.2.3 Compute: the work of learning

This section explains what training compute measures, why it grows with model size and data volume, and how its relationship to predictive performance depends on balancing those resources.

Key terms: compute, forward pass, loss function, backpropagation, latency, training compute, parameters, training tokens, forward and backward pass, model capacity

13.2.4 Expressing improvement as a power law

Explains how power laws describe declining language-model loss as resources grow, why the gains diminish, and where the empirical relationship may not hold.

Key terms: power-law distributions, scaling behavior, held-out loss, held-out cross-entropy loss, scaling resource, scaling exponent, proportional change, absolute reduction, diminishing returns, linear relationship

13.2.5 Balancing resources: the compute-efficient frontier

This section explains how to balance model size, training data, and compute to minimize prediction loss within a computational budget.

Key terms: training compute, parameters, tokens, parameter-limited, data-limited, undertrained, compute-efficient allocation, compute-efficient frontier, attainable loss, computational budget

13.3 What scaling revealed—and what it did not

This section distinguishes the measured benefits of scaling language models from the broader capabilities that lower loss alone does not establish.

Key terms: compute-efficient frontier, model capacity, training-data volume, held-out loss, held-out text, cross-entropy loss, next-token probabilities, broader capabilities

13.3.1 Larger models and sample efficiency

This section explains why larger language models can reach a fixed performance target with fewer training tokens, and why that does not necessarily reduce their total data or compute needs.

Key terms: sample efficiency, training tokens, held-out loss, model parameters, computational efficiency, training data, total compute

13.3.2 Generalization at high capacity

This section explains why larger language models can generalize despite their high capacity, while limiting that finding to the prediction tasks and data distributions tested.

Key terms: generalization, capacity, overfitting, held-out text, parameter count, data distributions

13.3.3 From next-token prediction to task transfer

The section explains how representations learned through next-token prediction can transfer to new tasks, how prompts enable in-context learning, and why these capabilities should not be attributed directly to Kaplan’s scaling-law experiments.

Key terms: transfer, next-token training, representations, prompts, in-context learning, few-shot learning, prompt, parameters, Kaplan’s experiments, language-model loss

13.3.4 The reach of the evidence

This section distinguishes what scaling experiments showed about neural language models from what their results cannot establish about capabilities, mechanisms, or intelligence.

Key terms: held-out cross-entropy loss, model parameters, training data, compute, next-token prediction, transfer, cross-entropy loss, predicted token probabilities, reliability, emergent capabilities

13.4 From isolated algorithms to scaled learning systems

This section connects observed improvements from scaling neural language models to the older foundations of learning under uncertainty.

Key terms: held-out loss, power-law relationships, model parameters, training data, compute, MacKay’s framework, uncertainty, constraints, representations, scaling

13.4.1 What scaling did not change

This section shows that scaling neural language models expands what learning systems can do without changing their basic task of reducing uncertainty from limited information under constraints.

Key terms: uncertainty, evidence, constraints, next token, preceding context, probability distribution, model capacity, scaling, useful structure

13.4.2 From learned representations to relational architecture

The section explains how neural networks and transformers expanded the ways learning systems use information while shifting, rather than eliminating, their constraints.

Key terms: internal representations, depth, representational capacity, recurrent neural networks, internal state, attention, contextual representations, parallel processing, uncertainty reduction under constraints, handcrafted representation

13.4.3 Scaling the reach of learning

This section explains how scaling extended the reach of existing learning principles and architectures, made some improvements predictable, and brought resource constraints into sharper focus.

Key terms: scaling, parameter spaces, training information, compute, predictive uncertainty, Kaplan, scaling laws, held-out loss, constraints, boundaries

13.5 Constraints pushed outward (not eliminated)

This section explains why predictable improvements from scaling face growing resource costs and physical, informational, and economic limits.

Key terms: power-law improvement, loss, resources, scaling laws, predictable improvement, sustainable improvement, physical, informational, and economic constraints, training data, performance gains

13.5.1 Computational and physical boundaries

This section explains why scaling frontier model training depends on memory, coordination, time, electricity, cooling, and infrastructure—not just faster processors or a larger compute budget.

Key terms: frontier scale, specialized accelerators, model capacity, training data, model parameters, intermediate activations, optimization states, memory bandwidth, communicated and synchronized, elapsed time

13.5.2 The limits of training data

This section explains why useful training information, rather than token count alone, limits scaling and how attempts to extend the data supply create new constraints.

Key terms: tokens, high-quality human-generated text, useful information, data volume, data quality, filtering, filtering and curation, synthetic data, multimodal data, validation

13.5.3 The economics of the frontier

This section explains how the rising costs of scaling can limit frontier AI development even when further technical progress remains possible.

Key terms: diminishing returns, loss, economic frontier, capital investment, training, inference, frontier-scale training, industrial infrastructure, economic boundary

13.5.4 Moving the boundary—again

The section explains how innovation can move the resource limits of scaling without eliminating constraints, and closes by identifying the enduring challenge shared by foundational algorithms and modern AI.

Key terms: compute-efficient frontier, parameters, data, compute, loss, continued scaling, new methods, greater resources, reduce uncertainty, available information

13.6 Summary

Concepts explained in this chapter

← Chapter 12: Beyond recurrence: Google Brain and the transformer architecture

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.