KV cache quantization reduces the precision of stored attention keys and values during autoregressive inference. Because attention’s dynamic memory consumption grows with context length, compressing the cache can make room for longer prompts or larger batches without changing model weights. section 8.0 section 8.7
Weight quantization and KV cache quantization are independent levers: the former reduces weight storage, while the latter targets dynamic cache growth. They can be combined in the same inference pipeline, but memory savings and throughput depend on the engine and GPU. section 8.1.2 section 8.7
A Hugging Face workflow isolates the cache’s effect:
- Load Llama-3.2-3B at its original weight precision and prepare the tokenizer and prompt. section 8.1.2
- Configure QuantizedCacheConfig with backend="HQQ", nbits=4, and the target CUDA device; pass it to generation with cache_implementation="quantized". section 8.1.2
- Measure static VRAM, dynamic VRAM growth, and throughput against an original-precision baseline and a 4-bit BitsAndBytes weight-quantized model. Inspect generated text as well. section 8.1.2 section 8.1.3
Listing 8.3 cache KV configuration
model_kvcache = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
torch_dtype=torch.float16, #1
device_map="auto"
)
cache_config = QuantizedCacheConfig( #2
backend="HQQ",
nbits=4,
device="cuda:0"
)
results_kvcache = measure_memory_allocation(
model_kvcache, tokenizer, PROMPT,
max_new_tokens=MAX_NEW_TOKENS,
cache_config=cache_config,
cache_implementation="quantized"
)
The configuration shows how to load original-precision weights and pass a 4-bit HQQ cache to the measurement helper. section 8.1.2
T4 and L4 comparison
For Llama-3.2-3B, the 4-bit HQQ cache reduced dynamic VRAM growth by about 3× relative to the original-precision and 4-bit weight configurations. On T4, an HQQ kernel reservation increased static VRAM; that increase was not observed on L4. The original-precision baseline had the best throughput, while both quantized configurations had a slight drop. In the shared 500-token prompt test, the cache-quantized response was coherent, whereas the weight-quantized response entered a repetition loop; this is one quality check, not a general guarantee. section 8.1.3
In vLLM, GPUs with native FP8 support doubled available KV token capacity without a throughput penalty in the reported comparison. On older T4 hardware, software overhead penalized generation speed. Benchmark the actual engine and hardware rather than assuming the same trade-off everywhere. section 8.7
Cache quantization can also be combined with attention-layer removal. To evaluate the combination, apply the same 4-bit HQQ configuration to a pruned model and compare memory, throughput, and generated-text quality against the full model with a quantized cache. section 8.5.2 section 8.6

