In neural language model scaling, diminishing returns means that held-out cross-entropy loss can keep declining as parameters, training data, or compute increase, while each further improvement requires a larger increase in resources. Power laws summarize this empirical pattern; they are not immutable rules, and their predictions depend on the conditions studied. section 13.2.4 section 13.1.3
Figure 13.1 Schematic scaling relationships. Held-out cross-entropy loss declines with model parameters, training-data volume, and compute when the other resources are not limiting. The approximately straight lines on logarithmic axes indicate power-law behavior across large changes in each resource. On standard axes, the same relationships flatten as resources increase: loss continues to decline, but progressively larger absolute increases in resources produce smaller absolute improvements.

The schematic shows loss curves flattening on standard axes: larger resource increases yield progressively smaller absolute improvements when other resources are not limiting. section 13.1.3
Diminishing returns also constrain decisions beyond the loss curve. Adding processors can yield smaller time savings as coordination grows more difficult, while further reductions in loss may not justify rising infrastructure and operating costs or deliver proportional practical value. section 13.5.1 section 13.5.3



