liveBook · Manning

Compute

As explained in Timeless Algorithms, The Foundational Ideas

A computational budget is the fixed amount of computation available for a task.

A fixed computational budget limits how much work can be spent training a model. For a given budget, increasing model size leaves less computation for processing training data; processing more tokens leaves less for a larger model. section 13.2.5

If N is the number of parameters and D the number of training tokens, training compute is approximately proportional to their product: C ∝ N × D. This is total training work, distinct from the time allowed for an individual prediction. Compute enables models to use their capacity and data, but cannot indefinitely compensate for insufficient capacity or uninformative data. section 13.2.3 section 13.2.5

A budget can be allocated poorly in either direction:

  • A model that is too small becomes parameter-limited: more training eventually yields little improvement because its capacity constrains what it can represent. section 13.2.5
  • A model that is too large for its budget may process too few tokens or receive too few updates, leaving its capacity underused. section 13.2.5
  • A compute-efficient allocation balances model size and training volume to minimize loss within the available budget. The best combination changes as the budget grows. section 13.2.5
Figure 13.2 Compute-efficient combinations of model size and training-data volume. Each curved contour represents a fixed computational budget. The bold line connects the combinations that produce the lowest attainable loss at each budget. Models above the frontier are relatively parameter-limited, while models below it are relatively data- or training-limited.

The frontier connects the model-size and training-data combinations that attain the lowest loss at each computational budget. section 13.2.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines training compute, gives its approximate parameter-token relationship, and explains balanced allocation without specifying an exact budget formula.
Examples
Contrasts total training work with real-time prediction in an autonomous vehicle and illustrates tradeoffs between parameters and training tokens.
What you can do after
Apply the compute-efficient frontier to choose model capacity and training-data volume for a fixed computational budget.
How it connects
Links model size and training data to compute, training compute to held-out loss, and resource balance to the compute-efficient frontier.

Also explained in

Compare how 4 books explain compute →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.