A network cannot learn patterns that its observations omit or represent poorly. Training data supplies the evidence from which a model learns patterns and forms internal representations. section 11.5.3 section 13.2.2
For language models, training data consists of tokens presented in sequences. These sequences show how words, facts, grammatical structures, and longer-range relationships occur in context. Greater volume and variety can expose models to more statistical regularities, while repetitive, inaccurate, or narrow data may add little new information. section 13.2.2
Data works alongside model capacity and compute: additional data helps only when a model has sufficient capacity and computation to use it. Under broadly comparable conditions, held-out loss declines predictably as training-data volume grows, with continuing but diminishing improvements. Capacity that grows faster than available evidence may remain underused or fit features of a limited sample too closely. section 13.1 section 13.2.2

















































