Scaling laws describe how training loss changes as parameters, data, and compute increase—but only within a particular training regime. A run can leave that regime when it runs out of data or its tokenizer, optimizer, or governing rule changes. section 6.2.1
On a log–log plot, these changes appear as joined straight-line segments rather than one universal line. Loss can remain smooth within a segment even as benchmark results jump: threshold-based evaluations may turn small improvements into many newly correct answers. section 6.2.1
Figure 6.4 A model will learn some tasks more easily than others. Easy tasks are frequent with dense data, while hard tasks are infrequent with sparse data.
Contrasts frequent, data-dense tasks with infrequent, data-sparse tasks, which models learn with different ease. section 6.2.1
