GPT-3 was not a radical architectural redesign of GPT-2. Its 175 billion parameters made it about 117 times larger, and it was trained on more data. The leap was backed by an empirical finding: language-model training loss tends to fall predictably as model size, training data, and compute increase. A scaling law describes that relationship. It is a way to forecast returns and allocate resources, not a promise that every capability will improve smoothly. section 6.0 section 6.2
An early clue came from Joel Hestness and colleagues’ 2017 study across vision, speech, and sequence models. Within a domain, error fell along a power-law curve as dataset size grew. Better architectures shifted the curve downward without changing its slope: they improved results at a given scale, while more data continued to produce gains at a characteristic rate. Jared Kaplan and colleagues’ 2020 language-model study extended this budgeting question to interacting limits on parameters, data, and compute. section 6.1 section 6.1
Why doubling one resource is not enough
In Kaplan’s estimates, doubling parameters reduced error by roughly 5%; doubling training data reduced it by roughly 6%. These are steady but diminishing gains, and either resource can become a bottleneck. A fixed dataset eventually stops a growing model from improving test loss and can invite overfitting. Kaplan estimated that an eightfold increase in parameters called for roughly five times as much data to maintain generalization. Under a fixed compute budget, the analysis favored a larger model trained for fewer steps over a smaller one trained to convergence. section 6.1.1 section 6.1.3
Figure 6.1 Language modeling performance improves smoothly as model size, dataset size, and amount of compute used for training increase. For optimal performance, all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each factor when it isn’t bottlenecked by the other two. Used with permission granted by a coauthor (Ben Cross).
The figure shows why a power-law gain from one scaling factor depends on the other two not becoming bottlenecks. section 6.1.1
That prescription was revised, not discarded. DeepMind’s 2022 Chinchilla study found that many large models had been undertrained: more tokens could make better use of their parameters. For transformers in that study’s setting, its compute-optimal estimate was about 20 training tokens per parameter. GPT-3’s roughly 1.7 tokens per parameter illustrated the mismatch. “Bigger” was no longer a sufficient training plan; the balance between size and data mattered. section 6.1.7
The sharpest caution is about what the curve measures. Training loss can decline smoothly while an exact-match evaluation suddenly jumps as many nearly correct answers cross a scoring threshold; other tasks can stagnate or worsen. A run can also enter a new regime when data runs short or training choices change, producing a kink rather than one universal line. Use scaling laws to plan compute and data, then use task evaluations to check what the resulting model can actually do. Whether the next gain justifies its marginal cost remains an engineering decision. section 6.2.1 section 6.2
