liveBook · Manning

Context window

As explained in Quantization and Fast Inference

Context length is the number of input tokens a language model can process as context for an inference.

A context window is the number of input tokens a model can process as context during inference. Longer contexts increase KV-cache memory traffic linearly: at 128K tokens, total traffic reaches 78 GB per token, 5.5× the level at 512 tokens. section 1.1

Figure 1.2 Memory traffic per token as context length grows. Model weights (filled) stay constant, but the KV cache (hatched) scales linearly with context. At 128K tokens, total memory traffic reaches 78 GB per token—5.5× more than at 512 tokens.

Compare how model-weight traffic stays constant while KV-cache traffic grows with context length. section 1.1

How Quantization and Fast Inference covers it

How deep it goes
Quantifies linear KV-cache growth and traffic at 128K versus 512 tokens, without detailing cache configuration.
Examples
Compares 128K-token contexts, at 78 GB of traffic per token, with 512-token contexts.
What you can do after
Understand how longer contexts increase KV-cache traffic and inference memory costs.
How it connects
Connects context length to KV cache, memory traffic, and inference costs.

Also explained in

Compare how 38 books explain context window →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.