A scaling law describes how model performance changes predictably as training resources increase. Common dimensions include model size, training data, and compute; larger models have often performed better on language tasks. section 2.13 section 2.7
Predictable improvement is not guaranteed for every ability. Some capabilities appear suddenly at larger sizes and do not follow established scaling patterns, making further gains difficult to forecast. More time spent reasoning during inference is another proposed route to better performance, though the overall contribution of scaling remains an open question. section 2.14 section 2.16 section 2.7
Capabilities that emerge at larger sizes
In reported evaluations, GPT-3 performed arithmetic close to random guessing at smaller sizes but answered addition, subtraction, and multiplication correctly more often at 13 billion parameters. Its performance on varied academic questions improved notably around 70 billion parameters or more. section 2.14
