liveBook · Manning

Key-value cache

As explained in Vibe Engineering

A KV cache is a stored collection of transformer attention keys and values from previously processed tokens that accelerates subsequent generation.

A key-value cache stores the attention keys and values for tokens already processed. Model weights stay fixed, but the cache grows linearly with context length, making the amount of text read a major local-inference memory constraint. section 9.3.6

Treat cache capacity as a deployment constraint rather than a rounding error:

  • Qwen 2.5 Coder 7B in half precision uses roughly 55 KB per token; 100,000 tokens require about 5–6 GB of cache, exceeding the roughly 4.5 GB Q4 weight footprint. section 9.3.6
  • A 14B model can require well over 20 GB for a full context in addition to about 9 GB of weights, putting large-context use beyond a 32 GB machine. section 9.3.6
  • Short tab-completion contexts keep the cache negligible, while interactive coding agents that read 100,000 or more tokens can make it rival or exceed model weights. section 9.3.6
  • Quantizing the cache to 8-bit or 4-bit reduces pressure with a possible quality cost; capping context avoids the cost by limiting what the model can read. section 9.3.6

A misleading fit check

A model whose weights fit in 40 GB may still fail to load a 100,000-token context, because the KV cache adds a separate, context-dependent memory requirement. section 9.3.4

Figure 9.5 Surprising cost structure.

Compare fixed weight memory with the context-dependent KV-cache cost to expose the surprising memory structure of local inference. section 9.3.6

How Vibe Engineering covers it

How deep it goes
Explains cache contents, linear growth, approximate memory costs, and mitigations, but does not detail attention-cache implementation.
Examples
Uses Qwen 2.5 Coder 7B, a 14B model, tab completion, and 100,000-token coding contexts as memory comparisons.
What you can do after
Evaluate local inference memory by accounting for context-dependent cache size alongside fixed model-weight requirements.
How it connects
Connects KV cache with transformer attention, context length, quantization, model weights, local inference, and cloud inference.

Also explained in

Compare how 9 books explain key-value cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.