A training distribution is the collection of examples and patterns a model learns from. A language model encounters the world indirectly through human-written text, so its capabilities are shaped by what that text frequently describes—and limited by what it rarely or never represents. section 6.2.1
Uneven capabilities
A model might handle multidigit arithmetic up to a certain number of digits but fail at the next, or generate a color-changing American flag website but not a cake that plays “Happy Birthday.” Similar-looking tasks can differ because their patterns are represented differently in training data. section 6.2.1
Figure 6.4 A model will learn some tasks more easily than others. Easy tasks are frequent with dense data, while hard tasks are infrequent with sparse data.
The figure contrasts tasks supported by frequent, dense examples with those supported by infrequent, sparse examples. section 6.2.1
