A scaling law describes how a system’s properties change as its resources vary. For neural language models, experiments reported in 2020 found that held-out prediction loss declines along relatively smooth power-law relationships as model parameters, training data, or training compute increase, provided other resources are not limiting. The law describes an empirical regularity, not a new learning mechanism. section 10.5.3 section 13.1.3
A simple power law can be written L ∝ R^−α, where L is loss, R is a resource, and α is a positive exponent describing the rate of improvement. On logarithmic axes, this relationship appears approximately as a straight line. On ordinary axes, it flattens: continued improvement requires progressively larger absolute increases in resources. section 13.1.3
Figure 13.1 Schematic scaling relationships. Held-out cross-entropy loss declines with model parameters, training-data volume, and compute when the other resources are not limiting. The approximately straight lines on logarithmic axes indicate power-law behavior across large changes in each resource. On standard axes, the same relationships flatten as resources increase: loss continues to decline, but progressively larger absolute increases in resources produce smaller absolute improvements.

Compare the same schematic scaling relationships on logarithmic and ordinary axes to see both the power-law pattern and diminishing returns. section 13.1.3
The three resources play distinct, interdependent roles:
- Parameters provide model capacity; training data supplies information from which patterns can be learned. section 10.5.3
- Compute performs the work of training. Under comparable conditions, training compute is approximately proportional to the number of parameters multiplied by the number of tokens processed. section 13.2.3
- No resource substitutes indefinitely for the others. A larger model needs sufficient data and computation, while more computation cannot supply information absent from the data. section 13.1.3 section 13.2.3
Estimating a larger training run
Results from smaller, less expensive runs can help estimate the loss attainable with additional resources before committing to a larger investment. Such estimates depend on the training conditions and ranges in which the observed relationship holds. section 13.1.3
Lower cross-entropy means better predicted token probabilities; it does not directly measure truthfulness, reasoning, or reliability. Larger models have also shown improved sample efficiency and generalization, and later systems transferred learned representations across language tasks. These observations do not establish that every capability will arise automatically or explain all the mechanisms behind the gains. section 13.3.4
Scaling remains constrained by data quality and availability, energy, memory, and economic cost. Changes in architecture, optimization, or data can alter the relationship, so extrapolation beyond observed conditions is uncertain. Scaling expands the reach of learning under constraints; it does not remove those constraints. section 10.5.3 section 13.1.3



