A context window is the maximum number of tokens an LLM can use as input while predicting the next token. The context_length setting defines this supported input size: longer text must be truncated. In a GPT pipeline, token IDs are embedded, positional embeddings are added, and the resulting sequence enters the model. section 2.8 section 5.1.1
Context size affects configuration, computation, and generation:
- Training can use a shorter window to reduce computational demands; a 256-token setting enables training on a standard laptop. section 5.1.1
- A GPT-2 model with 124 million parameters was originally configured for up to 1,024 tokens, while pretrained weights can later be loaded with that context length. section 5.1.1
- Positional embeddings are created for every supported position, so their length matches the configured context window. section 2.8
A configured generation window
Set GPT_CONFIG_124M["context_length"] to 256 and pass it as context_size to generate_text_simple. The model receives token IDs, produces logits, and decodes generated token IDs back into text. section 5.1.1
import torch
from chapter04 import GPTModel
GPT_CONFIG_124M = {
"vocab_size": 50257,
"context_length": 256, #1
"emb_dim": 768,
"n_heads": 12,
"n_layers": 12,
"drop_rate": 0.1, #2
"qkv_bias": False
}
torch.manual_seed(123)
model = GPTModel(GPT_CONFIG_124M)
model.eval()
Inspect the GPT configuration showing context_length alongside vocabulary size, embedding width, attention heads, and layers. section 5.1.1
