liveBook · Manning

Scaling law

A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.

How 4 Manning books explain it

Sutskever's List

A scaling law is an empirical relationship describing how model performance changes as resources such as parameters, data, or computation increase.

How it is used
Guides forecasts and allocation of parameters, training data, and compute for large language models.
How deep it goes
Explains empirical power-law trends, numerical trade-offs, and revisions to compute-optimal training without deriving the coupled equations.
Examples
Compares GPT-2 with GPT-3, parameter and data doubling, and GPT-3’s training-token ratio with Chinchilla’s estimate.
What you can do after
Evaluate/understand scaling forecasts while checking bottlenecks, changing regimes, task scores, and marginal cost.

Read how Sutskever's List explains scaling law →

Timeless Algorithms, The Foundational Ideas

A scaling law describes how a system’s properties change as its scale or resource levels vary.

How it is used
Uses scaling laws to forecast language-model performance and frame resource investment as an empirical route to improvement.
How deep it goes
Explains power-law loss trends, resource balance, and diminishing returns while distinguishing observed relationships from their unresolved mechanisms.
Examples
Uses schematic loss curves and estimates from smaller training runs to illustrate scaling and extrapolation.
What you can do after
Evaluate scaling forecasts by checking resource balance, extrapolation conditions, and the distinction between loss and broader capabilities.

Read how Timeless Algorithms, The Foundational Ideas explains scaling law →

Introduction to Generative AI, Second Edition

A scaling law is an empirical relationship describing how model performance changes as training resources, model size, data, compute, or inference-time effort increase.

How it is used
Describes how model performance changes with training scale and frames whether recent gains are predictable.
How deep it goes
Explains training and inference scaling at an overview level, while noting unpredictable emergent abilities and open questions.
Examples
Uses GPT-3 arithmetic and academic-question performance, plus o1's extended reasoning at inference time.
What you can do after
Evaluate whether performance gains appear predictable across model size, data, compute, and inference effort.

Read how Introduction to Generative AI, Second Edition explains scaling law →

Build a DeepSeek Model (From Scratch)

A scaling law is an empirical relationship describing how a model's performance or other properties change as resources such as model size increase.

How it is used
Uses empirical performance trends to guide model-size and base-model choices in knowledge distillation.
How deep it goes
Explains observed size trends and base-model interactions, then identifies teacher, student, and dataset scaling as open questions.
Examples
Compares Qwen2.5-Math, Qwen2.5, and Llama-3 distilled models using AIME 2024 performance across parameter sizes.
What you can do after
Evaluate how model size and base-model capabilities affect distilled-model performance.

Read how Build a DeepSeek Model (From Scratch) explains scaling law →

Also covered in

What to know first

Related concepts

Questions about scaling law

What is scaling law?

A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.

Which Manning books explain scaling law?

Sutskever's List, Timeless Algorithms, The Foundational Ideas, Introduction to Generative AI, Second Edition and Build a DeepSeek Model (From Scratch).

What should I know before learning scaling law?

Manning books that teach scaling law usually introduce language model, transformer architecture, generalization and GPT-4 first.

What is the difference between scaling law and No Free Lunch Theorem?

A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary. The No Free Lunch theorems state that, averaged across all possible problems, no learning or optimization algorithm is universally superior to every other algorithm.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.