Scaling law
A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.
Sutskever's List
A scaling law is an empirical relationship describing how model performance changes as resources such as parameters, data, or computation increase.
- How it is used
- Guides forecasts and allocation of parameters, training data, and compute for large language models.
- How deep it goes
- Explains empirical power-law trends, numerical trade-offs, and revisions to compute-optimal training without deriving the coupled equations.
- Examples
- Compares GPT-2 with GPT-3, parameter and data doubling, and GPT-3’s training-token ratio with Chinchilla’s estimate.
- What you can do after
- Evaluate/understand scaling forecasts while checking bottlenecks, changing regimes, task scores, and marginal cost.
Timeless Algorithms, The Foundational Ideas
A scaling law describes how a system’s properties change as its scale or resource levels vary.
- How it is used
- Uses scaling laws to forecast language-model performance and frame resource investment as an empirical route to improvement.
- How deep it goes
- Explains power-law loss trends, resource balance, and diminishing returns while distinguishing observed relationships from their unresolved mechanisms.
- Examples
- Uses schematic loss curves and estimates from smaller training runs to illustrate scaling and extrapolation.
- What you can do after
- Evaluate scaling forecasts by checking resource balance, extrapolation conditions, and the distinction between loss and broader capabilities.
Read how Timeless Algorithms, The Foundational Ideas explains scaling law →
Introduction to Generative AI, Second Edition
A scaling law is an empirical relationship describing how model performance changes as training resources, model size, data, compute, or inference-time effort increase.
- How it is used
- Describes how model performance changes with training scale and frames whether recent gains are predictable.
- How deep it goes
- Explains training and inference scaling at an overview level, while noting unpredictable emergent abilities and open questions.
- Examples
- Uses GPT-3 arithmetic and academic-question performance, plus o1's extended reasoning at inference time.
- What you can do after
- Evaluate whether performance gains appear predictable across model size, data, compute, and inference effort.
Read how Introduction to Generative AI, Second Edition explains scaling law →
Build a DeepSeek Model (From Scratch)
A scaling law is an empirical relationship describing how a model's performance or other properties change as resources such as model size increase.
- How it is used
- Uses empirical performance trends to guide model-size and base-model choices in knowledge distillation.
- How deep it goes
- Explains observed size trends and base-model interactions, then identifies teacher, student, and dataset scaling as open questions.
- Examples
- Compares Qwen2.5-Math, Qwen2.5, and Llama-3 distilled models using AIME 2024 performance across parameter sizes.
- What you can do after
- Evaluate how model size and base-model capabilities affect distilled-model performance.
Read how Build a DeepSeek Model (From Scratch) explains scaling law →
Also covered in
- Deep Learning with Python, Third Edition
- Deep Learning with R, Third Edition
- Machines that ThinkIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- generalizationGeneralization is the extension of learned patterns, abstractions, or conclusions from observed cases to new cases beyond those from which they were derived.
- GPT-4GPT-4 is a multimodal large language model developed by OpenAI that processes text and image inputs and generates text.
Related concepts
- dataset sizeuses · Dataset size is the number of examples, records, or other data items contained in or available from a dataset.
- power-law relationshipuses · A power-law relationship is a mathematical relationship in which one quantity is proportional to another quantity raised to a fixed exponent.
- token-to-parameter ratioproduces · The number of training tokens processed per learned model parameter.
- emergent behaviorcontrasts with · System-level behavior or capability that arises from interactions among components or from a system’s scale or complexity, rather than being specified as an isolated outcome.
- objective functionmeasures · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Questions about scaling law
What is scaling law?
A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.
Which Manning books explain scaling law?
Sutskever's List, Timeless Algorithms, The Foundational Ideas, Introduction to Generative AI, Second Edition and Build a DeepSeek Model (From Scratch).
What should I know before learning scaling law?
Manning books that teach scaling law usually introduce language model, transformer architecture, generalization and GPT-4 first.
What is the difference between scaling law and No Free Lunch Theorem?
A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary. The No Free Lunch theorems state that, averaged across all possible problems, no learning or optimization algorithm is universally superior to every other algorithm.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



