A key-value cache (KV cache) stores attention key and value tensors from previously processed tokens. During autoregressive generation, each new token can attend to the stored tensors instead of recomputing them. The speed benefit comes at a memory cost: the cache grows with sequence length and can exhaust GPU memory on long contexts. section 3.1.2 section 4.1.2
Cache size depends on the number of attention layers, KV heads, head dimension, tokens, batch size, and bytes per element. A useful estimate is: KV bytes = 2 × attention layers × KV heads × head dimension × sequence length × bytes per element × batch size. The factor of two accounts for keys and values; engine allocation can introduce differences from the estimate. section 12.7.2 section 12.6.2
Listing 12.5 KV cache footprint before and after pruning
def kv_cache_bytes(L_attn, H_kv, d_head, seq_len, batch, dtype_bytes=2):
return 2 * L_attn * H_kv * d_head * seq_len * batch * dtype_bytes
base = kv_cache_bytes(L_attn=28, H_kv=8, d_head=128,
seq_len=2048, batch=64) #1
slim = kv_cache_bytes(L_attn=16, H_kv=8, d_head=128,
seq_len=2048, batch=64) #2
print(f"cache after pruning is {slim / base:.1%} of the original")Use the footprint function to compare a 28-attention-layer model with a 16-attention-layer configuration under the same workload. section 12.7.2
Three ways to reduce cache memory address different terms in the estimate:
- Share KV heads: grouped-query attention shares each KV pair across a group of query heads; multi-query attention shares one pair across all query heads. Fewer KV heads mean fewer cached tensors per token. section 3.1.2
- Reduce precision: cache quantization compresses keys and values before writing them and dequantizes them for attention. It saves memory but loses precision and can impose a speed penalty depending on the engine and GPU. section 8.1.1
- Remove attention layers: depth pruning or attention removal reduces the layers contributing entries. This saves memory but can degrade model quality; attention removal can also require custom serving code and create a non-uniform cache. section 4.1.2 section 12.5
Check an allocation against the estimate
For Llama-3.2-3B in float16, 28 attention blocks, 8 KV heads, and a head dimension of 128 give 114,688 bytes per cached token. On a Colab T4, vLLM reported 5.63 GiB for 52,720 tokens, approximately 114,666 bytes per token; the small difference reflects rounding. The same run generated at 33.55 ms per token at batch size 1, a baseline rather than a general speed claim. section 12.6.2








