A sliding window moves a fixed-length window across tokenized text to create overlapping input–target pairs for next-token prediction. Each input contains a token sequence, while its target sequence is shifted one position to the right. section 2.6
A typical implementation follows this data flow:
- Tokenize the text and represent it as integer token IDs. section 2.6
- Slice an input chunk of max_length tokens and create its target chunk by shifting the slice one position. section 2.6
- Advance the next window by stride positions; stride 1 creates overlap, while stride equal to the window size prevents overlap. section 2.6
- Collect the pairs in a PyTorch Dataset and return batches through DataLoader. section 2.6
Shifted context and target
With max_length 4, an input such as [290, 4920, 2241, 287] receives [4920, 2241, 287, 257] as its targets. Moving the window by stride 1 makes the next input begin with the previous input’s second token. section 2.6
Figure 2.12 Given a text sample, extract input blocks as subsamples that serve as input to the LLM, and the LLM’s prediction task during training is to predict the next word that follows the input block. During training, we mask out all words that are past the target. Note that the text shown in this figure must undergo tokenization before the LLM can process it; however, this figure omits the tokenization step for clarity.
Shows how input blocks become next-token prediction tasks, including the portion beyond the target that is masked during training. section 2.6
