liveBook · Manning

Key-value cache

As explained in Rearchitecting LLMs

A key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.

A key-value cache (KV cache) stores attention key and value tensors from previously processed tokens. During autoregressive generation, each new token can attend to the stored tensors instead of recomputing them. The speed benefit comes at a memory cost: the cache grows with sequence length and can exhaust GPU memory on long contexts. section 3.1.2 section 4.1.2

Cache size depends on the number of attention layers, KV heads, head dimension, tokens, batch size, and bytes per element. A useful estimate is: KV bytes = 2 × attention layers × KV heads × head dimension × sequence length × bytes per element × batch size. The factor of two accounts for keys and values; engine allocation can introduce differences from the estimate. section 12.7.2 section 12.6.2

Listing 12.5 KV cache footprint before and after pruning
def kv_cache_bytes(L_attn, H_kv, d_head, seq_len, batch, dtype_bytes=2):
    return 2 * L_attn * H_kv * d_head * seq_len * batch * dtype_bytes
 
base = kv_cache_bytes(L_attn=28, H_kv=8, d_head=128, 
                      seq_len=2048, batch=64) #1
slim = kv_cache_bytes(L_attn=16, H_kv=8, d_head=128, 
                      seq_len=2048, batch=64) #2
print(f"cache after pruning is {slim / base:.1%} of the original")

Use the footprint function to compare a 28-attention-layer model with a 16-attention-layer configuration under the same workload. section 12.7.2

Three ways to reduce cache memory address different terms in the estimate:

  • Share KV heads: grouped-query attention shares each KV pair across a group of query heads; multi-query attention shares one pair across all query heads. Fewer KV heads mean fewer cached tensors per token. section 3.1.2
  • Reduce precision: cache quantization compresses keys and values before writing them and dequantizes them for attention. It saves memory but loses precision and can impose a speed penalty depending on the engine and GPU. section 8.1.1
  • Remove attention layers: depth pruning or attention removal reduces the layers contributing entries. This saves memory but can degrade model quality; attention removal can also require custom serving code and create a non-uniform cache. section 4.1.2 section 12.5

Check an allocation against the estimate

For Llama-3.2-3B in float16, 28 attention blocks, 8 KV heads, and a head dimension of 128 give 114,688 bytes per cached token. On a Colab T4, vLLM reported 5.63 GiB for 52,720 tokens, approximately 114,666 bytes per token; the small difference reflects rounding. The same run generated at 33.55 ms per token at batch size 1, a baseline rather than a general speed claim. section 12.6.2

How Rearchitecting LLMs covers it

How deep it goes
Explains reuse, footprint arithmetic, quantization cycles, and pruning trade-offs, then checks an engine allocation against the estimate.
Examples
Compares 28 versus 16 attention layers and checks a Llama-3.2-3B vLLM allocation on a Colab T4.
What you can do after
Evaluate cache footprint and memory trade-offs using layer count, KV heads, precision, sequence length, and batch size.
How it connects
Connects the cache to grouped-query attention, quantization, depth pruning, attention pruning, and paged attention.

Also explained in

Compare how 9 books explain key-value cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.