liveBook · Manning

Training dataset

As explained in Timeless Algorithms, The Foundational Ideas

Training data is the dataset used to fit a model.

A network cannot learn patterns that its observations omit or represent poorly. Training data supplies the evidence from which a model learns patterns and forms internal representations. section 11.5.3 section 13.2.2

For language models, training data consists of tokens presented in sequences. These sequences show how words, facts, grammatical structures, and longer-range relationships occur in context. Greater volume and variety can expose models to more statistical regularities, while repetitive, inaccurate, or narrow data may add little new information. section 13.2.2

Data works alongside model capacity and compute: additional data helps only when a model has sufficient capacity and computation to use it. Under broadly comparable conditions, held-out loss declines predictably as training-data volume grows, with continuing but diminishing improvements. Capacity that grows faster than available evidence may remain underused or fit features of a limited sample too closely. section 13.1 section 13.2.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines data as learning evidence and relates its volume to held-out loss, while treating data quality as a separate constraint.
Examples
Uses language tokens in sequences, including examples of words, facts, grammar, and longer-range relationships.
What you can do after
Understand how data volume, variety, and quality constrain learning and model-scaling gains.
How it connects
Connects training data to model capacity, compute, learned representations, and held-out loss.

Also explained in

Compare how 50 books explain training dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.