A context window is the maximum number of tokens a language model can look at in one pass. Think of it as the model’s temporary reading space: a 32-token window can use the latest 32 tokens, but cannot see an entire novel at once. section 11.10 section 11.11
The context window matters because it:
- Limits each training or inference input to a manageable token sequence. section 11.10
- Determines how much preceding text the Transformer can use when building contextual meaning. section 11.11
- Acts as a model hyperparameter alongside vocabulary size, width, depth, and attention heads. section 11.11 section 11.35
Turning a long stream into batches
After tokenization, a long document becomes one continuous stream of token IDs. If the context-window size is 32, the stream is divided into 32-token training examples so the model can process one batch at a time. A batching routine takes a start index, slices the next 32 tokens, appends that sequence, and continues until the remaining stream is too short for a complete window. section 11.10
