liveBook · Manning

Kv cache quantization

KV cache quantization is the use of reduced numerical precision to store attention keys and values in a model’s key-value cache.

How 2 Manning books explain it

Rearchitecting LLMs

KV cache quantization reduces the numerical precision of stored attention key and value states to lower memory use during autoregressive inference.

How it is used
Reduces dynamic attention memory during autoregressive inference to support longer prompts or larger batches.
How deep it goes
Provides a Hugging Face configuration and measurement workflow, with comparative results and an exercise combining cache quantization with attention pruning.
Examples
Compares Llama-3.2-3B configurations on T4 and L4 GPUs using static VRAM, dynamic VRAM, throughput, and generated text.
What you can do after
Apply KV cache quantization via Hugging Face and HQQ, then compare memory use, throughput, and output quality against baselines.

Read how Rearchitecting LLMs explains kv cache quantization →

Quantization and Fast Inference

KV cache quantization is the reduced-precision representation of the attention keys and values stored for prior tokens.

How it is used
Reduces long-context KV cache memory pressure to increase LLM serving capacity.
How deep it goes
Explains cache sizing, repeated reads, outlier channels, and attention sensitivity, but stops short of a quantized cache implementation.
Examples
Calculates Llama-3.1-70B cache size and compares FP16 versus INT4 concurrency for Llama-3.1-8B on an A100.
What you can do after
Evaluate cache precision choices using memory estimates, attention sensitivity, and repeated dequantization costs.

Read how Quantization and Fast Inference explains kv cache quantization →

What to know first

Related concepts

Questions about kv cache quantization

What is kv cache quantization?

KV cache quantization is the use of reduced numerical precision to store attention keys and values in a model’s key-value cache.

Which Manning books explain kv cache quantization?

Rearchitecting LLMs and Quantization and Fast Inference.

What should I know before learning kv cache quantization?

Manning books that teach kv cache quantization usually introduce autoregressive generation, context window, key-value cache and memory usage first.

What is the difference between kv cache quantization and structural modification?

KV cache quantization is the use of reduced numerical precision to store attention keys and values in a model’s key-value cache. A structural modification is a modification of a model's architecture that physically removes or bypasses components.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.