Emergent behavior describes capabilities that appear when a model reaches a certain scale, though smaller models perform near chance. GPT-3 made the idea striking: without a reengineered architecture, it demonstrated in-context learning and performed tasks without task-specific fine-tuning. section 6.0
The apparent suddenness is partly a measurement effect. Training loss can improve smoothly while thresholded evaluations—such as exact-match scores—jump when borderline answers become correct. Regime changes in data, tokenizers, or optimizers can also bend scaling trends and coincide with shifts in ability. section 6.2.1 section 6.2.1
Reported examples and cautions include:
- Multistep reasoning, multidigit arithmetic, translation between languages not explicitly trained on, and basic code generation. section 6.0
- Capabilities do not scale uniformly: some tasks plateau or decline, while others improve sharply after flat periods. Benchmarks sample only part of the possible input space. section 6.2 section 6.2.1
GPT-3 and the scaling puzzle
GPT-3 had about 117 times GPT-2’s parameters. Scaling laws had suggested that training loss would decrease predictably as models, data, and compute grew; its in-context learning and other task abilities nevertheless seemed to arrive abruptly. section 6.0
Figure 6.4 A model will learn some tasks more easily than others. Easy tasks are frequent with dense data, while hard tasks are infrequent with sparse data.
The figure contrasts easier tasks, associated with frequent and dense data, with harder tasks associated with sparse data. section 6.2.1
