The leap from GPT-2 to GPT-3 made model scale hard to ignore. GPT-2, released in 2019, had 1.5 billion parameters. A year later, OpenAI released the 175-billion-parameter GPT-3. The researchers did not reengineer the architecture; they scaled the model and its training data, backed by substantial engineering infrastructure. GPT-3 showed in-context learning and competence on tasks without task-specific fine-tuning. section 6.0 section 1.1
A convincing sample is not a benchmark
GPT-2’s “Ovid’s Unicorn” captured attention with its convincing tone and style. But the sample was cherry-picked and contained an awkward reference to “four-horned unicorns.” It illustrates why an impressive piece of generated text cannot, by itself, establish reliable performance. section 6.0
Scaling laws helped explain the bet on GPT-3: training loss decreases smoothly as model size, data, and compute increase. Some benchmarked abilities, such as arithmetic, translation, and code generation, can nevertheless appear suddenly at larger scales. That apparent emergence may partly reflect how performance is measured, rather than an abrupt change in the underlying loss curve. section 6.0
Bigger was not necessarily the best use of a training budget. GPT-3 was trained on 300 billion tokens, about 1.7 tokens per parameter. Chinchilla’s analysis argued that, for its era, a roughly 20:1 ratio would use parameters more effectively: GPT-3 could instead have been much smaller or trained on far more tokens. GPT-3.5 later became the first model behind ChatGPT, while the data-versus-size trade-off continued to shape comparisons among large models. section 6.1.7 section 1.1
