Scaling law
A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.
Sutskever's List
A scaling law is an empirical relationship describing how model performance changes as resources such as parameters, data, or computation increase.
- How it is used
- Guides forecasts and allocation of parameters, training data, and compute for large language models.
- How deep it goes
- Explains empirical power-law trends, numerical trade-offs, and revisions to compute-optimal training without deriving the coupled equations.
- Examples
- Compares GPT-2 with GPT-3, parameter and data doubling, and GPT-3’s training-token ratio with Chinchilla’s estimate.
- What you can do after
- Evaluate/understand scaling forecasts while checking bottlenecks, changing regimes, task scores, and marginal cost.
Introduction to Generative AI, Second Edition
A scaling law is an empirical relationship describing how model performance changes as training resources, model size, data, compute, or inference-time effort increase.
- How it is used
- Describes how model performance changes with training scale and frames whether recent gains are predictable.
- How deep it goes
- Explains training and inference scaling at an overview level, while noting unpredictable emergent abilities and open questions.
- Examples
- Uses GPT-3 arithmetic and academic-question performance, plus o1's extended reasoning at inference time.
- What you can do after
- Evaluate whether performance gains appear predictable across model size, data, compute, and inference effort.
Read how Introduction to Generative AI, Second Edition explains scaling law →
Related concepts
- power-law relationshipuses · A power-law relationship is a mathematical relationship in which one quantity is proportional to another quantity raised to a fixed exponent.
- token-to-parameter ratioproduces · The number of training tokens processed per learned model parameter.
- dataset sizeuses · Dataset size is the number of examples, records, or other data items contained in or available from a dataset.
- objective functionmeasures · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
- emergent behaviorcontrasts with · System-level behavior or capability that arises from interactions among components or from a system’s scale or complexity, rather than being specified as an isolated outcome.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.