A downstream task tests a model’s practical capability after training. Training-loss scaling can be predicted precisely, but performance on tasks such as arithmetic, coding, or planning may rise unevenly; general benchmarks offer a better, though limited, view. section 6.2.1
Uneven performance
A model might generate a color-changing American flag but fail to make a cake that plays “Happy Birthday,” or know how to tie one shoe but not the other. Benchmark accuracy can also decline temporarily as models scale before recovering. section 6.2.1
Figure 6.4 A model will learn some tasks more easily than others. Easy tasks are frequent with dense data, while hard tasks are infrequent with sparse data.
Illustrates why tasks may be learned unevenly: easier tasks occur more often in denser data, while harder tasks are less frequent. section 6.2.1
