Kv cache quantization
KV cache quantization is the use of reduced numerical precision to store attention keys and values in a model’s key-value cache.
Rearchitecting LLMs
KV cache quantization reduces the numerical precision of stored attention key and value states to lower memory use during autoregressive inference.
- How it is used
- Reduces dynamic attention memory during autoregressive inference to support longer prompts or larger batches.
- How deep it goes
- Provides a Hugging Face configuration and measurement workflow, with comparative results and an exercise combining cache quantization with attention pruning.
- Examples
- Compares Llama-3.2-3B configurations on T4 and L4 GPUs using static VRAM, dynamic VRAM, throughput, and generated text.
- What you can do after
- Apply KV cache quantization via Hugging Face and HQQ, then compare memory use, throughput, and output quality against baselines.
Read how Rearchitecting LLMs explains kv cache quantization →
Quantization and Fast Inference
KV cache quantization is the reduced-precision representation of the attention keys and values stored for prior tokens.
- How it is used
- Reduces long-context KV cache memory pressure to increase LLM serving capacity.
- How deep it goes
- Explains cache sizing, repeated reads, outlier channels, and attention sensitivity, but stops short of a quantized cache implementation.
- Examples
- Calculates Llama-3.1-70B cache size and compares FP16 versus INT4 concurrency for Llama-3.1-8B on an A100.
- What you can do after
- Evaluate cache precision choices using memory estimates, attention sensitivity, and repeated dequantization costs.
Read how Quantization and Fast Inference explains kv cache quantization →
What to know first
- autoregressive generationAutoregressive generation produces a sequence one element at a time, conditioning each new element on the elements already available.
- context windowA context window is a bounded span of sequence elements or contextual information that a model or processing operation can consider together.
- key-value cacheA key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- memory usageMemory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
Related concepts
- attentionaddresses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- structural modificationcontrasts with · A structural modification is a modification of a model's architecture that physically removes or bypasses components.
- Hugging Faceuses · Hugging Face is a platform and ecosystem for hosting, discovering, sharing, and using machine-learning models, datasets, applications, and related tools.
- vLLMuses · vLLM is an open-source inference and serving engine for large language models, designed for high-throughput and memory-efficient execution.
- weight-only quantizationcontrasts with · Weight-only quantization reduces the precision used to represent model weights while retaining higher precision for activations.
Questions about kv cache quantization
What is kv cache quantization?
KV cache quantization is the use of reduced numerical precision to store attention keys and values in a model’s key-value cache.
Which Manning books explain kv cache quantization?
Rearchitecting LLMs and Quantization and Fast Inference.
What should I know before learning kv cache quantization?
Manning books that teach kv cache quantization usually introduce autoregressive generation, context window, key-value cache and memory usage first.
What is the difference between kv cache quantization and structural modification?
KV cache quantization is the use of reduced numerical precision to store attention keys and values in a model’s key-value cache. A structural modification is a modification of a model's architecture that physically removes or bypasses components.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

