liveBook · Manning

Power-law relationship

As explained in Timeless Algorithms, The Foundational Ideas

A power-law relationship is one in which a quantity varies as a power of another quantity.

A power-law relationship describes how one quantity changes in proportion to a fixed power of another. In language-model scaling, it can summarize how held-out loss changes as parameters, training data, or compute increase, provided the other resources are not limiting. section 13.1.3

A simplified form is L(X) = aX^(−b), where L is held-out cross-entropy loss, X is a scaling resource, a sets the scale, and b controls the rate of decline. Increasing X by a factor k changes loss by a factor k^(−b): L(kX) = k^(−b)L(X). Thus, the same proportional increase in resources yields the same proportional reduction in loss. section 13.2.4

Continuing improvement, diminishing absolute returns

For L(X) = X^(−0.1), loss falls from 1.00 to about 0.79 as X rises from 1 to 10, then to 0.63 at 100 and 0.50 at 1,000. Each tenfold resource increase produces a similar proportional reduction, but the absolute reduction becomes smaller. section 13.2.4

Figure 13.1 Schematic scaling relationships. Held-out cross-entropy loss declines with model parameters, training-data volume, and compute when the other resources are not limiting. The approximately straight lines on logarithmic axes indicate power-law behavior across large changes in each resource. On standard axes, the same relationships flatten as resources increase: loss continues to decline, but progressively larger absolute increases in resources produce smaller absolute improvements.

The schematic contrasts approximately straight scaling trends on logarithmic axes with flattening curves on standard axes. section 13.1.3

On logarithmic axes, a power law appears as a straight line whose slope indicates the rate of change. Straightness alone is not conclusive: log-normal and other nonlinear processes can look similar over a limited range. Some observed language-model trends extended across more than seven orders of magnitude, strengthening the evidence for stable scaling relationships. section 13.1.3

A scaling relationship describes how two quantities vary together; it is distinct from a power-law distribution, such as a distribution of word frequencies. The scaling relationships do not imply that model resources themselves follow power-law distributions. section 13.2.4

These laws are empirical regularities, not immutable rules. Smaller runs can inform estimates for larger investments, but predictions depend on the architectures, data, training conditions, and constraints under which the relationships were measured. Data volume alone does not determine data quality, and lower loss does not guarantee factual accuracy or reliability. section 13.1.3 section 13.2.4 section 13.2.2 section 13.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Derives the effect of proportional resource increases from a simplified equation and interprets log-log plots, while warning that apparent straightness is inconclusive.
Examples
Uses word frequencies to contrast distributions with scaling and calculates loss at four resource levels under an exponent of −0.1.
What you can do after
Evaluate power-law scaling claims by interpreting proportional changes, resource constraints, and the limits of log-log evidence.
How it connects
Connects power-law scaling to held-out cross-entropy loss, model capacity, training data, compute, and diminishing returns.

Also explained in

Compare how 2 books explain power-law relationship →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.