vLLM is an inference engine for production-oriented large language model serving. It uses PagedAttention and reserves a fixed share of GPU memory at startup for KV-cache blocks; the number of tokens those blocks can hold constrains context capacity and batch size. section 8.1 section 8.2
Tune memory use with two distinct levers:
- Quantize the KV cache to FP8 to reduce the size of its blocks and fit more tokens in the reserved pool. section 8.2.1
- Quantize model weights to NF4 to reduce their VRAM footprint, leaving more of the fixed budget for KV-cache blocks. section 8.2
- Measure both KV-token capacity and generation throughput; 4-bit cache quantization is avoided in this workflow because its extra GPU calculations can penalize latency. section 8.2.1
T4 baseline
A float16 Llama-3.2-3B control run on a Colab T4 reserved 5.63 GiB for KV cache, holding 52,720 tokens, and generated at 33.55 milliseconds per token. These figures provide a comparison point for that setup, not a general performance guarantee. section 12.6.2
Figure 8.4 VRAM allocation across inference engines and quantization strategies. In Hugging Face, the KV cache grows dynamically into available memory. In vLLM, a fixed memory budget is divided at startup between model weights and a pre-allocated KV block pool. Reducing weight precision to NF4 shrinks the weight footprint, expanding the KV block pool within the same budget. Quantizing the KV blocks themselves reduces each block's size, fitting more tokens into the same pool.
Compare how weight and cache quantization change the memory available for vLLM's pre-allocated KV block pool. section 8.2



