A key-value (KV) cache stores attention keys and values from prior tokens so an autoregressive transformer can reuse them instead of recomputing them at each generation step. The cache is written as tokens arrive and reread during generation, so longer contexts increase both its memory footprint and memory traffic. section 1.1 section 3.0
Runtime memory also includes model weights and a compute buffer. Unlike the weights, the KV cache grows linearly with context length. Its size depends on the number of decoder layers, KV heads, head dimension, context length, and bytes per cached value. section 10.5.1
For FP16 keys and values, cache bytes = 2 × layers × KV heads × head dimension × context length × 2 bytes. The first factor of 2 accounts for keys and values; the last accounts for the two bytes per FP16 value. Multi-head attention uses one KV head per attention head, while grouped-query attention shares KV projections across query heads. section 10.5.1
Sizing a 4,096-token cache
Llama-2-7B uses 32 layers, 32 KV heads and a head dimension of 128: 2 × 32 × 32 × 128 × 4,096 × 2 = 2,048 MiB. Llama-3-8B has the same layer count and head dimension but eight KV heads, reducing the cache to 512 MiB at the same context length. section 10.5.1
Figure 10.11 Runtime memory footprint for Q4_K_M Llama-2-7B (MHA, 32 KV heads) and Q4_K_M Llama-3-8B (GQA, 8 KV heads) at 2048 and 4096 context. Weights are nearly fixed across context lengths; the KV cache (/// hatching) scales linearly with context and dominates the footprint at long ctx for the MHA model. The compute buffer (xxx hatching referred as comp in the fig), activation scratch the runtime allocates per forward pass, sits around 100 MiB for Llama-2-7B and grows to 266 MiB for Llama-3-8B because activation scratch on the output side scales with vocabulary, and Llama-3's 128K vocabulary is 4× Llama-2's. Llama-3-8B's 4× reduction in KV heads (GQA) cuts the KV cache footprint by 4× at the same context: the Llama-3-8B 4096-ctx bar's KV segment is 512 MiB versus 1024 MiB for the Llama-2-7B 2048-ctx bar despite holding twice as many tokens of state. Numbers measured from llama-cli stderr on c3-standard-22, llama.cpp build at commit 6d57a49.

Compare cache, weight, and compute-buffer footprints at two context lengths for multi-head and grouped-query attention. section 10.5.1
Cache quantization is a separate decision from weight and activation quantization: their statistics and error sensitivities differ, and keys and values have a structural asymmetry. Precision choices must account for accuracy, tensor statistics, and hardware. At long context, the cache can equal the model in size. section 3.0 section 7.0
In serving, memory freed by smaller weights can expand the KV pool and potentially support more concurrent sequences before requests queue. Check occupancy and queue depth rather than assuming a throughput benefit: a 3B-model sweep on a 24 GB card kept occupancy below about 1.3% and queue depth at zero for both FP16 and AWQ weights. section 12.3.2








