A context window is the number of input tokens a model can process as context during inference. Longer contexts increase KV-cache memory traffic linearly: at 128K tokens, total traffic reaches 78 GB per token, 5.5× the level at 512 tokens. section 1.1
Figure 1.2 Memory traffic per token as context length grows. Model weights (filled) stay constant, but the KV cache (hatched) scales linearly with context. At 128K tokens, total memory traffic reaches 78 GB per token—5.5× more than at 512 tokens.

Compare how model-weight traffic stays constant while KV-cache traffic grows with context length. section 1.1





































