liveBook · Manning

Kv cache quantization

As explained in Rearchitecting LLMs

KV cache quantization reduces the numerical precision of stored attention key and value states to lower memory use during autoregressive inference.

KV cache quantization reduces the precision of stored attention keys and values during autoregressive inference. Because attention’s dynamic memory consumption grows with context length, compressing the cache can make room for longer prompts or larger batches without changing model weights. section 8.0 section 8.7

Weight quantization and KV cache quantization are independent levers: the former reduces weight storage, while the latter targets dynamic cache growth. They can be combined in the same inference pipeline, but memory savings and throughput depend on the engine and GPU. section 8.1.2 section 8.7

A Hugging Face workflow isolates the cache’s effect:

  • Load Llama-3.2-3B at its original weight precision and prepare the tokenizer and prompt. section 8.1.2
  • Configure QuantizedCacheConfig with backend="HQQ", nbits=4, and the target CUDA device; pass it to generation with cache_implementation="quantized". section 8.1.2
  • Measure static VRAM, dynamic VRAM growth, and throughput against an original-precision baseline and a 4-bit BitsAndBytes weight-quantized model. Inspect generated text as well. section 8.1.2 section 8.1.3
Listing 8.3 cache KV configuration
model_kvcache = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16, #1
    device_map="auto"
)
 
cache_config = QuantizedCacheConfig(  #2
    backend="HQQ",
    nbits=4,
    device="cuda:0"
)
 
results_kvcache = measure_memory_allocation(
    model_kvcache, tokenizer, PROMPT,
    max_new_tokens=MAX_NEW_TOKENS,
    cache_config=cache_config,        
    cache_implementation="quantized"
)

The configuration shows how to load original-precision weights and pass a 4-bit HQQ cache to the measurement helper. section 8.1.2

T4 and L4 comparison

For Llama-3.2-3B, the 4-bit HQQ cache reduced dynamic VRAM growth by about 3× relative to the original-precision and 4-bit weight configurations. On T4, an HQQ kernel reservation increased static VRAM; that increase was not observed on L4. The original-precision baseline had the best throughput, while both quantized configurations had a slight drop. In the shared 500-token prompt test, the cache-quantized response was coherent, whereas the weight-quantized response entered a repetition loop; this is one quality check, not a general guarantee. section 8.1.3

In vLLM, GPUs with native FP8 support doubled available KV token capacity without a throughput penalty in the reported comparison. On older T4 hardware, software overhead penalized generation speed. Benchmark the actual engine and hardware rather than assuming the same trade-off everywhere. section 8.7

Cache quantization can also be combined with attention-layer removal. To evaluate the combination, apply the same 4-bit HQQ configuration to a pruned model and compare memory, throughput, and generated-text quality against the full model with a quantized cache. section 8.5.2 section 8.6

How Rearchitecting LLMs covers it

How deep it goes
Provides a Hugging Face configuration and measurement workflow, with comparative results and an exercise combining cache quantization with attention pruning.
Examples
Compares Llama-3.2-3B configurations on T4 and L4 GPUs using static VRAM, dynamic VRAM, throughput, and generated text.
What you can do after
Apply KV cache quantization via Hugging Face and HQQ, then compare memory use, throughput, and output quality against baselines.
How it connects
Connects KV cache quantization with weight quantization, attention pruning, and autoregressive generation.

Also explained in

Compare how 2 books explain kv cache quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.