A key-value cache stores the attention keys and values for tokens already processed. Model weights stay fixed, but the cache grows linearly with context length, making the amount of text read a major local-inference memory constraint. section 9.3.6
Treat cache capacity as a deployment constraint rather than a rounding error:
- Qwen 2.5 Coder 7B in half precision uses roughly 55 KB per token; 100,000 tokens require about 5–6 GB of cache, exceeding the roughly 4.5 GB Q4 weight footprint. section 9.3.6
- A 14B model can require well over 20 GB for a full context in addition to about 9 GB of weights, putting large-context use beyond a 32 GB machine. section 9.3.6
- Short tab-completion contexts keep the cache negligible, while interactive coding agents that read 100,000 or more tokens can make it rival or exceed model weights. section 9.3.6
- Quantizing the cache to 8-bit or 4-bit reduces pressure with a possible quality cost; capping context avoids the cost by limiting what the model can read. section 9.3.6
A misleading fit check
A model whose weights fit in 40 GB may still fail to load a 100,000-token context, because the KV cache adds a separate, context-dependent memory requirement. section 9.3.4
Figure 9.5 Surprising cost structure.

Compare fixed weight memory with the context-dependent KV-cache cost to expose the surprising memory structure of local inference. section 9.3.6








