Next-token prediction trains a language model to infer the next word or token from preceding context. The task uses language’s sequential structure to learn context, structure, and relationships without manually collected labels. section 1.1 section 1.6
GPT applies the objective autoregressively: a decoder-only transformer processes text left to right, predicts one next token, and can use each generated output as context for the following prediction. Pretraining on this simple objective supports later text completion, classification, spelling correction, translation, and instruction fine-tuning. section 1.6 section 1.8
A practical training pipeline constructs aligned context and target tensors:
- Tokenize the source text into token IDs. section 2.6
- Create an input sequence x and a target sequence y by shifting the same token stream one position: y contains the next token for every position in x. section 2.6
- Extract fixed-length windows with a configurable stride, convert them to tensors, and batch them with a PyTorch DataLoader. section 2.6
Shifted context and target
With context_size = 4, x = [290, 4920, 2241, 287] and y = [4920, 2241, 287, 257]. The progressively growing contexts therefore produce [290] → 4920, [290, 4920] → 2241, and so on. A data loader returns these aligned tensors in batches.
