An input-target pair is a tokenized training example for next-token prediction: the target sequence is the input sequence shifted one position forward. The input shows the context; each target gives the token the model should predict next. section 2.6
A sliding-window dataset constructs pairs as follows:
- Tokenize the text into token IDs, then select an input window of length max_length. section 2.6
Four-token window
For x = [290, 4920, 2241, 287], the target is y = [4920, 2241, 287, 257]. The sequence therefore trains four predictions: [290] → 4920, [290, 4920] → 2241, [290, 4920, 2241] → 287, and the full window → 257.
