liveBook · Manning

Key-value cache

As explained in Quantization and Fast Inference

Runtime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.

A key-value (KV) cache stores attention keys and values from prior tokens so an autoregressive transformer can reuse them instead of recomputing them at each generation step. The cache is written as tokens arrive and reread during generation, so longer contexts increase both its memory footprint and memory traffic. section 1.1 section 3.0

Runtime memory also includes model weights and a compute buffer. Unlike the weights, the KV cache grows linearly with context length. Its size depends on the number of decoder layers, KV heads, head dimension, context length, and bytes per cached value. section 10.5.1

For FP16 keys and values, cache bytes = 2 × layers × KV heads × head dimension × context length × 2 bytes. The first factor of 2 accounts for keys and values; the last accounts for the two bytes per FP16 value. Multi-head attention uses one KV head per attention head, while grouped-query attention shares KV projections across query heads. section 10.5.1

Sizing a 4,096-token cache

Llama-2-7B uses 32 layers, 32 KV heads and a head dimension of 128: 2 × 32 × 32 × 128 × 4,096 × 2 = 2,048 MiB. Llama-3-8B has the same layer count and head dimension but eight KV heads, reducing the cache to 512 MiB at the same context length. section 10.5.1

Figure 10.11 Runtime memory footprint for Q4_K_M Llama-2-7B (MHA, 32 KV heads) and Q4_K_M Llama-3-8B (GQA, 8 KV heads) at 2048 and 4096 context. Weights are nearly fixed across context lengths; the KV cache (/// hatching) scales linearly with context and dominates the footprint at long ctx for the MHA model. The compute buffer (xxx hatching referred as comp in the fig), activation scratch the runtime allocates per forward pass, sits around 100 MiB for Llama-2-7B and grows to 266 MiB for Llama-3-8B because activation scratch on the output side scales with vocabulary, and Llama-3's 128K vocabulary is 4× Llama-2's. Llama-3-8B's 4× reduction in KV heads (GQA) cuts the KV cache footprint by 4× at the same context: the Llama-3-8B 4096-ctx bar's KV segment is 512 MiB versus 1024 MiB for the Llama-2-7B 2048-ctx bar despite holding twice as many tokens of state. Numbers measured from llama-cli stderr on c3-standard-22, llama.cpp build at commit 6d57a49.

Compare cache, weight, and compute-buffer footprints at two context lengths for multi-head and grouped-query attention. section 10.5.1

Cache quantization is a separate decision from weight and activation quantization: their statistics and error sensitivities differ, and keys and values have a structural asymmetry. Precision choices must account for accuracy, tensor statistics, and hardware. At long context, the cache can equal the model in size. section 3.0 section 7.0

In serving, memory freed by smaller weights can expand the KV pool and potentially support more concurrent sequences before requests queue. Check occupancy and queue depth rather than assuming a throughput benefit: a 3B-model sweep on a 24 GB card kept occupancy below about 1.3% and queue depth at zero for both FP16 and AWQ weights. section 12.3.2

How Quantization and Fast Inference covers it

How deep it goes
Derives FP16 cache memory from architecture and context length, then compares serving occupancy and queue depth.
Examples
Compares Llama-2-7B and Llama-3-8B cache sizes and an FP16 versus AWQ serving sweep.
What you can do after
Evaluate KV-cache memory requirements and serving headroom using architecture parameters, occupancy, and queue depth.
How it connects
Connects cache sizing to grouped-query attention, quantization, memory bandwidth, and request scheduling.

Also explained in

Compare how 9 books explain key-value cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.