PagedAttention manages key-value (KV) cache memory during inference. In vLLM, a fixed share of VRAM is preallocated for KV blocks at engine startup, so reducing model-weight memory can increase the pool's token capacity. Quantizing the KV blocks can fit more tokens into that pool. section 8.2
Figure 8.4 VRAM allocation across inference engines and quantization strategies. In Hugging Face, the KV cache grows dynamically into available memory. In vLLM, a fixed memory budget is divided at startup between model weights and a pre-allocated KV block pool. Reducing weight precision to NF4 shrinks the weight footprint, expanding the KV block pool within the same budget. Quantizing the KV blocks themselves reduces each block's size, fitting more tokens into the same pool.
Compare dynamic KV-cache growth with vLLM's preallocated block pool, and see how weight and KV-cache quantization change its capacity. section 8.2
Check savings on the target engine
Removing attention from Transformer blocks reduces KV-cache size, but uniform per-layer pages may not fit a model with heterogeneous KV state cleanly. Verify the predicted savings on the real engine: one pinned version reported 90,058 bytes per token against a prediction of 90,112. section 12.7.2


