A training dataset is the text subset used to fit a language model’s parameters. For GPT-style pretraining, it supplies diverse token sequences from which the model learns syntax, semantics, context, and some general knowledge. The raw corpus becomes usable only after tokenization and organization into prediction examples. section 1.5 section 2.6
A practical training-data pipeline proceeds as follows:
- Tokenize the text into token IDs; a 20,479-character story produces 5,145 tokens in the worked dataset. section 2.6 section 5.1.3
- Create an input sequence x and a target sequence y by shifting the same token stream one position, so each target is the next token. section 2.6
- Slide a fixed-length window through the tokens using a stride; stride 1 creates overlap, while stride equal to the window length prevents overlap. section 2.6
- Batch the resulting tensors with a PyTorch DataLoader, optionally shuffling rows and dropping incomplete training batches. section 2.6
- Split text into training and validation portions before preparing loaders, then compare cross-entropy losses during model evaluation. section 5.1.3
Four-token next-token example
With context_size = 4, x = [290, 4920, 2241, 287] and y = [4920, 2241, 287, 257]. The model therefore receives each growing context and predicts the token immediately to its right, such as “and established himself in” → “a.” section 2.6
Table 1.1 The pretraining dataset of the popular GPT-3 LLM
|
Dataset name
|
Dataset description
|
Number of tokens
|
Proportion in training data
|
|---|---|---|---|
|
CommonCrawl (filtered)
|
Web crawl data
|
410 billion
|
60%
|
|
WebText2
|
Web crawl data
|
19 billion
|
22%
|
|
Books1
|
Internet-based book corpus
|
12 billion
|
8%
|
|
Books2
|
Internet-based book corpus
|
55 billion
|
8%
|
|
Wikipedia
|
High-quality text
|
3 billion
|
3%
|
A dataset-composition table shows how a large GPT-3 pretraining corpus combines filtered web pages, books, and Wikipedia, including token counts and proportions. section 1.5
Figure 2.12 Given a text sample, extract input blocks as subsamples that serve as input to the LLM, and the LLM’s prediction task during training is to predict the next word that follows the input block. During training, we mask out all words that are past the target. Note that the text shown in this figure must undergo tokenization before the LLM can process it; however, this figure omits the tokenization step for clarity.
A sliding-window figure clarifies how text blocks become next-token prediction tasks and how tokens beyond the current target are masked. section 2.6
