liveBook · Manning

Kv cache quantization

As explained in Quantization and Fast Inference

KV cache quantization is the reduced-precision representation of the attention keys and values stored for prior tokens.

KV cache quantization stores attention keys and values from prior tokens in reduced precision. Its main benefit is memory savings during generation, especially at long context lengths and larger batch sizes; it is not primarily a compute-speedup technique. section 3.1 section 7.1

Each generation step computes keys and values for the new token and appends them to the cache. Existing entries are written once but read repeatedly on subsequent steps. Quantization therefore reduces stored data, while dequantization overhead can recur many times. section 3.1

For L layers, H cached attention heads, head dimension D, sequence length S, and batch size B, the cache contains 2 × L × H × D × S × B elements. The factor of two accounts for keys and values. Memory grows linearly with sequence length, batch size, and bytes per element. section 3.1

Long-context memory

For Llama-3.1-70B, an FP16 cache at 128K context occupies 40 GB for one request. At batch size 8, it reaches 320 GB, more than twice the stated model-weight size. section 3.1

Listing 3.7 KV cache size calculation
def kv_cache_size_gb(num_layers, num_heads, head_dim, seq_length, 
                      batch_size, bytes_per_element=2):
    total_bytes = 2 * num_layers * num_heads * head_dim * \
                  seq_length * batch_size * bytes_per_element    #A
    return total_bytes / (1024**3)
 
llama_70b = dict(num_layers=80, num_heads=8, head_dim=128)
model_size_fp16 = 140  # GB
for seq_len in [2048, 8192, 32768, 131072]:
    kv_size = kv_cache_size_gb(**llama_70b, seq_length=seq_len, batch_size=1)
    ratio = kv_size / model_size_fp16 * 100
    print(f"Seq {seq_len:>6}: {kv_size:>6.2f} GB ({ratio:>5.1f}% of weights)")
 

The size calculation makes the dependence on architecture, context length, batch size, and precision explicit. section 3.1

Precision choice must account for attention error as well as capacity. Some hidden dimensions consistently produce larger key or value magnitudes, suggesting per-channel quantization. A small key error can shift the softmax attention distribution over all values, so reduced cache precision can affect output accuracy. section 3.1

Serving capacity on an A100

For Llama-3.1-8B with INT4 weights on an 80 GB A100, an estimate leaves 74 GB for the cache after weights and overhead. At 128K context, FP16 KV uses 16 GB per user and permits four users; INT4 KV uses 4 GB per user and permits 18. This is a capacity estimate, not a throughput measurement. section 7.1

Table 7.1 Concurrent users on an 80GB A100 serving Llama-3.1-8B (INT4 weights, ~4 GB model)
Context length
FP16 KV per user
Max users
INT4 KV per user
Max users
Multiplier
2,048
0.25 GB
296
0.06 GB
1,184
4.0×
8,192
1.00 GB
74
0.25 GB
296
4.0×
32,768
4.00 GB
18
1.00 GB
74
4.1×
131,072
16.00 GB
4
4.00 GB
18
4.5×

The comparison shows how the estimated concurrent-user limit changes with context length and KV precision. section 7.1

Estimate cache memory for the intended context and batch before selecting precision. Consider per-channel treatment for persistent outliers, then weigh accuracy and repeated dequantization cost against the capacity gained. Cache savings can complement weight quantization because both reduce GPU memory pressure. section 3.1 section 7.1

How Quantization and Fast Inference covers it

How deep it goes
Explains cache sizing, repeated reads, outlier channels, and attention sensitivity, but stops short of a quantized cache implementation.
Examples
Calculates Llama-3.1-70B cache size and compares FP16 versus INT4 concurrency for Llama-3.1-8B on an A100.
What you can do after
Evaluate cache precision choices using memory estimates, attention sensitivity, and repeated dequantization costs.
How it connects
Connects per-channel quantization, outlier channels, softmax attention, and weight quantization to cache precision.

Also explained in

Compare how 2 books explain kv cache quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.