A training dataset is the collection of examples used to build or optimize a machine-learning model. For language models, it is largely human-produced text gathered into a training corpus. The model adjusts itself toward outputs that resemble those examples. section 4.2 section 1.6
The dataset supplies the target pattern, but not necessarily the goal people care about. A language model is rewarded for producing text that resembles its training data, not for being truthful, original, or factually accurate. If the data contains errors, stereotypes, or bad code, matching those patterns can reinforce them. section 4.2
Training datasets shape a model through several practical properties:
- Scale matters: modern language models may use hundreds of gigabytes of internet-scraped text, while larger data pools can improve many tasks. section 1.6 section 4.2
- Coverage matters: the data can include websites, news articles, research papers, government reports, code, and mathematics rather than only books. section 9.3
- Quality matters: incorrect or oddly represented information can make a model more likely to reproduce false common knowledge, social bias, or recurring bugs. section 4.2
- Provenance matters: much training data comes from sources whose licenses do not clearly permit AI training, leaving important legal and ethical questions unresolved. section 9.3
Why resemblance is not truth
Suppose the training data repeatedly contains a false claim or buggy programming pattern. Producing similar text can reduce the model’s training loss, so the model may become better at reproducing the error even while becoming worse at truth or correctness. The same pattern applies to stereotypes and social biases found in the data. section 4.2
