KV cache quantization stores attention keys and values from prior tokens in reduced precision. Its main benefit is memory savings during generation, especially at long context lengths and larger batch sizes; it is not primarily a compute-speedup technique. section 3.1 section 7.1
Each generation step computes keys and values for the new token and appends them to the cache. Existing entries are written once but read repeatedly on subsequent steps. Quantization therefore reduces stored data, while dequantization overhead can recur many times. section 3.1
For L layers, H cached attention heads, head dimension D, sequence length S, and batch size B, the cache contains 2 × L × H × D × S × B elements. The factor of two accounts for keys and values. Memory grows linearly with sequence length, batch size, and bytes per element. section 3.1
Long-context memory
For Llama-3.1-70B, an FP16 cache at 128K context occupies 40 GB for one request. At batch size 8, it reaches 320 GB, more than twice the stated model-weight size. section 3.1
Listing 3.7 KV cache size calculation
def kv_cache_size_gb(num_layers, num_heads, head_dim, seq_length,
batch_size, bytes_per_element=2):
total_bytes = 2 * num_layers * num_heads * head_dim * \
seq_length * batch_size * bytes_per_element #A
return total_bytes / (1024**3)
llama_70b = dict(num_layers=80, num_heads=8, head_dim=128)
model_size_fp16 = 140 # GB
for seq_len in [2048, 8192, 32768, 131072]:
kv_size = kv_cache_size_gb(**llama_70b, seq_length=seq_len, batch_size=1)
ratio = kv_size / model_size_fp16 * 100
print(f"Seq {seq_len:>6}: {kv_size:>6.2f} GB ({ratio:>5.1f}% of weights)")
The size calculation makes the dependence on architecture, context length, batch size, and precision explicit. section 3.1
Precision choice must account for attention error as well as capacity. Some hidden dimensions consistently produce larger key or value magnitudes, suggesting per-channel quantization. A small key error can shift the softmax attention distribution over all values, so reduced cache precision can affect output accuracy. section 3.1
Serving capacity on an A100
For Llama-3.1-8B with INT4 weights on an 80 GB A100, an estimate leaves 74 GB for the cache after weights and overhead. At 128K context, FP16 KV uses 16 GB per user and permits four users; INT4 KV uses 4 GB per user and permits 18. This is a capacity estimate, not a throughput measurement. section 7.1
Table 7.1 Concurrent users on an 80GB A100 serving Llama-3.1-8B (INT4 weights, ~4 GB model)
|
Context length
|
FP16 KV per user
|
Max users
|
INT4 KV per user
|
Max users
|
Multiplier
|
|
2,048
|
0.25 GB
|
296
|
0.06 GB
|
1,184
|
4.0×
|
|
8,192
|
1.00 GB
|
74
|
0.25 GB
|
296
|
4.0×
|
|
32,768
|
4.00 GB
|
18
|
1.00 GB
|
74
|
4.1×
|
|
131,072
|
16.00 GB
|
4
|
4.00 GB
|
18
|
4.5×
|
The comparison shows how the estimated concurrent-user limit changes with context length and KV precision. section 7.1
Estimate cache memory for the intended context and batch before selecting precision. Consider per-channel treatment for persistent outliers, then weigh accuracy and repeated dequantization cost against the capacity gained. Cache savings can complement weight quantization because both reduce GPU memory pressure. section 3.1 section 7.1

