A fixed computational budget limits how much work can be spent training a model. For a given budget, increasing model size leaves less computation for processing training data; processing more tokens leaves less for a larger model. section 13.2.5
If N is the number of parameters and D the number of training tokens, training compute is approximately proportional to their product: C ∝ N × D. This is total training work, distinct from the time allowed for an individual prediction. Compute enables models to use their capacity and data, but cannot indefinitely compensate for insufficient capacity or uninformative data. section 13.2.3 section 13.2.5
A budget can be allocated poorly in either direction:
- A model that is too small becomes parameter-limited: more training eventually yields little improvement because its capacity constrains what it can represent. section 13.2.5
- A model that is too large for its budget may process too few tokens or receive too few updates, leaving its capacity underused. section 13.2.5
- A compute-efficient allocation balances model size and training volume to minimize loss within the available budget. The best combination changes as the budget grows. section 13.2.5
Figure 13.2 Compute-efficient combinations of model size and training-data volume. Each curved contour represents a fixed computational budget. The bold line connects the combinations that produce the lowest attainable loss at each budget. Models above the frontier are relatively parameter-limited, while models below it are relatively data- or training-limited.

The frontier connects the model-size and training-data combinations that attain the lowest loss at each computational budget. section 13.2.5



