liveBook · Manning

vLLM

As explained in Rearchitecting LLMs

vLLM is an open-source library and inference engine for efficient large language model serving.

vLLM is an inference engine for production-oriented large language model serving. It uses PagedAttention and reserves a fixed share of GPU memory at startup for KV-cache blocks; the number of tokens those blocks can hold constrains context capacity and batch size. section 8.1 section 8.2

Tune memory use with two distinct levers:

  • Quantize the KV cache to FP8 to reduce the size of its blocks and fit more tokens in the reserved pool. section 8.2.1
  • Quantize model weights to NF4 to reduce their VRAM footprint, leaving more of the fixed budget for KV-cache blocks. section 8.2
  • Measure both KV-token capacity and generation throughput; 4-bit cache quantization is avoided in this workflow because its extra GPU calculations can penalize latency. section 8.2.1

T4 baseline

A float16 Llama-3.2-3B control run on a Colab T4 reserved 5.63 GiB for KV cache, holding 52,720 tokens, and generated at 33.55 milliseconds per token. These figures provide a comparison point for that setup, not a general performance guarantee. section 12.6.2

Figure 8.4 VRAM allocation across inference engines and quantization strategies. In Hugging Face, the KV cache grows dynamically into available memory. In vLLM, a fixed memory budget is divided at startup between model weights and a pre-allocated KV block pool. Reducing weight precision to NF4 shrinks the weight footprint, expanding the KV block pool within the same budget. Quantizing the KV blocks themselves reduces each block's size, fitting more tokens into the same pool.

Compare how weight and cache quantization change the memory available for vLLM's pre-allocated KV block pool. section 8.2

How Rearchitecting LLMs covers it

How deep it goes
Implements cache and weight quantization experiments, measures token capacity and throughput, and discusses hardware trade-offs.
Examples
Tests FP8 cache and NF4 weight options, and compares a float16 Llama-3.2-3B baseline on a Colab T4.
What it assumes
Builds on familiarity with model-weight quantization and explains KV-cache quantization by analogy.
What you can do after
Operate vLLM with quantization options and measure KV-token capacity and generation throughput for a configured GPU.
How it connects
Connects PagedAttention and KV-cache quantization with GPU memory allocation, model-weight quantization, and Hugging Face inference.

Also explained in

Compare how 4 books explain vLLM →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.