A training dataset is a collection of examples used to train a machine-learning model. For a large language model (LLM), those examples are chiefly text. Developers assemble enormous collections of writing so the model can learn patterns of language and generate natural-looking responses. The amount of text matters, but so does what it contains. section 1.12
Common sources offer different benefits and risks:
- Wikipedia provides human-edited text in many languages; digitized public-domain books provide published prose, though some may be outdated or inaccurate. section 1.12
- Reddit provides examples of conversation, but can also contain slurs, misinformation, and other content a developer may not want the model to reproduce. section 1.12
Dataset choices shape more than fluency. Internet text can carry stereotypes and historical inequalities, while uneven representation can leave some communities poorly reflected in a model’s outputs. Older data can also leave a model out of step with changing public discussions. These problems can arise alongside bias introduced during labeling or deployment; they are not all caused by the dataset alone. section 3.2
When a prompt may become training data
An attorney who pastes a client contract into a dialogue agent to request revisions may expose confidential details if prompts are reviewed or used for training. Whether that happens depends on the service’s practices. The possibility makes data-use policies and client consent important before sharing sensitive text. section 3.10
Privacy risks also begin with the original collection. A generative model can sometimes reproduce sensitive text encountered during training, including information gathered from the web. Avoiding sources known to contain substantial personally identifiable information, seeking permission, and protecting sensitive data can reduce risk, but detecting or removing every private detail is difficult. Public availability alone does not mean people expect their posts to be used for AI training. section 10.18 section 10.9
