A dataset is a collection of examples arranged for model training or experimentation. In a GPT pipeline, it becomes tokenized input–target pairs: each input context contains a token sequence, while its target sequence is shifted one position to represent next-token prediction. section 2.6
Prepare a dataset by turning raw text or labeled records into uniform batches:
- Tokenize raw text and generate overlapping or non-overlapping windows with a configurable maximum length and stride. section 2.6
- Store inputs and shifted targets as tensors; a PyTorch Dataset exposes individual pairs through __len__ and __getitem__. section 2.6
- Balance labeled data when one class dominates, map labels to integers, and split records into training, validation, and test sets. section 6.2
- Pad variable-length messages to a shared maximum length, then use DataLoader to batch, shuffle, and optionally drop incomplete batches. section 6.3
Next-token and classification batches
With context_size=4, an input such as [290, 4920, 2241, 287] pairs with the target [4920, 2241, 287, 257]. For classification, a batch can contain eight padded messages of 120 token IDs and eight labels. section 2.6 section 6.3
Figure 2.12 Given a text sample, extract input blocks as subsamples that serve as input to the LLM, and the LLM’s prediction task during training is to predict the next word that follows the input block. During training, we mask out all words that are past the target. Note that the text shown in this figure must undergo tokenization before the LLM can process it; however, this figure omits the tokenization step for clarity.
Shows how tokenized input blocks define the next-token prediction targets and mask later positions during training. section 2.6
