Next-token prediction trains a language model to predict the token that follows a sequence. Each training example pairs an input sequence with the same sequence shifted one position to the right, turning language modeling into multicategory classification over the vocabulary. section 8.3.2 section 10.3
A typical training pipeline proceeds in small, repeatable steps:
- Tokenize text and map each unique token to an integer index. section 12.1
- Cut the indexed text into fixed-length input sequences. section 8.3.2 section 11.1.1
- Shift every input one token to the right to create its target sequence. section 8.3.2
- Train the model so each position predicts the corresponding target token using the available context. section 11.1.1
A one-token shift
For the input sequence "how are you", the target sequence is "are you today". The model learns to predict "are" after "how", "you" after "are", and so forth, gradually learning a probability distribution over possible next tokens. section 8.3.2
Listing 8.2 Creating training data
import torch
seq_len=100 #1
xys=[]
for n in range(0, len(wordidx)-seq_len-1): #2
x = wordidx[n:n+seq_len] #3
y = wordidx[n+1:n+seq_len+1] #4
xys.append((torch.tensor(x),(torch.tensor(y))))
The compact PyTorch listing makes the alignment explicit: x selects a sequence, while y selects the same window beginning at the next token. section 8.3.2
During generation, the trained model predicts one likely next token, appends it to the sequence, and repeats the process. Recurrent networks and decoder-only Transformers use this objective; Transformers prevent access to future tokens with causal self-attention. section 12.1 section 11.1.1
