liveBook · Manning

Diminishing returns

As explained in Timeless Algorithms, The Foundational Ideas

The principle that successive increases in an input produce progressively smaller incremental gains in an outcome.

In neural language model scaling, diminishing returns means that held-out cross-entropy loss can keep declining as parameters, training data, or compute increase, while each further improvement requires a larger increase in resources. Power laws summarize this empirical pattern; they are not immutable rules, and their predictions depend on the conditions studied. section 13.2.4 section 13.1.3

Figure 13.1 Schematic scaling relationships. Held-out cross-entropy loss declines with model parameters, training-data volume, and compute when the other resources are not limiting. The approximately straight lines on logarithmic axes indicate power-law behavior across large changes in each resource. On standard axes, the same relationships flatten as resources increase: loss continues to decline, but progressively larger absolute increases in resources produce smaller absolute improvements.

The schematic shows loss curves flattening on standard axes: larger resource increases yield progressively smaller absolute improvements when other resources are not limiting. section 13.1.3

Diminishing returns also constrain decisions beyond the loss curve. Adding processors can yield smaller time savings as coordination grows more difficult, while further reductions in loss may not justify rising infrastructure and operating costs or deliver proportional practical value. section 13.5.1 section 13.5.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains power-law improvement and its limits, including conditional predictions and resource, infrastructure, and economic constraints.
Examples
Illustrates flattening loss curves and describes diminishing time savings from adding processors.
What you can do after
Evaluate whether expected loss reductions justify the rising resource and operating costs of further scaling.
How it connects
Links diminishing returns to power laws, scaling laws, held-out cross-entropy loss, and compute parallelism.

Also explained in

Compare how 4 books explain diminishing returns →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.