An input-output pair is a training example for next-token prediction: x contains a token sequence, while y contains the same sequence shifted one token to the right. Each position in x therefore supplies context for the target token at the matching position in y. section 12.1.2 section 12.2.2
From a sentence to targets
For “the old man and the sea,” x contains “the old man and the,” while y contains “old man and the sea.” The model learns to predict “old” from “the,” “man” from “the old,” and so on. With a sequence length of 128, each pair contains 128 token indexes; batches of 32 produce x and y tensors shaped (32, 128). section 12.1.2 section 12.2.2
Construct pairs by following the sequence window and training workflow:
- Choose a fixed sequence length, such as 128 indexes. section 12.2.2
- Set x to the current window and y to the window beginning at the next token. section 12.2.2
- Collect the pairs, shuffle them into batches, and feed x through the GPT model. section 12.2.2 section 12.1.2
- Compare predictions with y using cross-entropy loss, then adjust model parameters. section 12.1.2
import torch
seq_len=128 #1
xys=[]
for n in range(0, len(wordidx)-seq_len-1):
x = wordidx[n:n+seq_len] #2
y = wordidx[n+1:n+seq_len+1] #3
xys.append((torch.tensor(x),(torch.tensor(y)))) #4
The construction listing shows how a sliding token window creates each x, y pair as PyTorch tensors. section 12.2.2
