GPT-2 made an awkward point for researchers looking for a new algorithm: its techniques were familiar, but its scale produced strikingly fluent prose. Scaling means increasing a model’s parameters, training data, or computation to improve performance. The result was impressive without being dependable: GPT-2 could sustain a convincing article’s form while inventing impossible details. Its capabilities also prompted OpenAI to withhold the full model initially, turning a technical advance into a release-policy dispute. section 1.2
Scale is a multiplier, not a substitute for a workable method. Before deep learning’s rise, skeptics argued that larger datasets, rather than neural-network architecture, explained apparent gains; in one image-localization comparison, a neural network did no better than nearest neighbors on the same data. Later architectures made a different case. Residual connections helped signals pass through deep networks, while attention removed a sequential training bottleneck. More data and compute became valuable because these methods could put them to work. section 2.2 section 3.8 section 5.8
Scaling laws made the trade-offs measurable. In experiments with language models, loss improved along smooth curves as model size or training data grew, though each increase brought diminishing returns. Holding parameter count constant while changing a model’s shape—such as using fewer wide layers instead of more thin ones—changed loss relatively little except at extreme shapes. The question shifted from whether to grow a model to how to divide resources among parameters, data, and compute. section 6.1.2 section 6.1.3
Two influential recipes illustrate why the balance matters:
- Kaplan’s 2020 results suggested increasing the dataset by roughly fivefold for each eightfold increase in parameters to maintain generalization. This made very large models seem feasible without matching every parameter increase with proportional data growth. section 6.1.3
- DeepMind’s 2022 Chinchilla analysis found that many large models were undertrained. For the transformers it studied, roughly 20 training tokens per parameter was compute-optimal; a smaller, better-trained model could beat a larger one starved of data. section 6.1.7
When bigger is not better trained
GPT-3 had 175 billion parameters and was trained on 300 billion tokens—about 1.7 tokens per parameter. Under the Chinchilla analysis, it could instead have been much smaller or trained on far more tokens to use its parameters more effectively. The comparison does not make parameter count irrelevant; it shows why parameter count alone is a poor measure of how well training resources were spent. section 6.1.7
Smooth improvement curves are not a promise that every increase helps under any conditions. Double descent describes a rise in test error near the point where a model fits its training data, followed by a fall as capacity grows further. Balancing data and model size can keep training away from that troublesome region. Nor does lower loss settle broader claims about intelligence: interpretations of scaling ranged from the scaling hypothesis to critiques of apparent reasoning. Larger pretrained models still did not deliver dependable multistep reasoning or planning. section 8.5.3 section 6.1.6 section 6.4
Figure 8.10 Test (solid line) and training (dashed line) errors for ResNet as the width increases. A peak near the interpolation threshold (dashed line; shaded “critical” band) illustrates double descent, in which error rises as the model fits the data, then falls again for larger widths. Used with permission granted by the lead author (Preetum Nakkiran).
The ResNet error curves show why scaling can briefly look worse near the interpolation threshold before larger widths improve test error. section 8.5.3
