liveBook · Manning

PagedAttention

As explained in Rearchitecting LLMs

PagedAttention is an attention-memory management technique that stores key-value cache blocks in pages to improve memory utilization during inference.

PagedAttention manages key-value (KV) cache memory during inference. In vLLM, a fixed share of VRAM is preallocated for KV blocks at engine startup, so reducing model-weight memory can increase the pool's token capacity. Quantizing the KV blocks can fit more tokens into that pool. section 8.2

Figure 8.4 VRAM allocation across inference engines and quantization strategies. In Hugging Face, the KV cache grows dynamically into available memory. In vLLM, a fixed memory budget is divided at startup between model weights and a pre-allocated KV block pool. Reducing weight precision to NF4 shrinks the weight footprint, expanding the KV block pool within the same budget. Quantizing the KV blocks themselves reduces each block's size, fitting more tokens into the same pool.

Compare dynamic KV-cache growth with vLLM's preallocated block pool, and see how weight and KV-cache quantization change its capacity. section 8.2

Check savings on the target engine

Removing attention from Transformer blocks reduces KV-cache size, but uniform per-layer pages may not fit a model with heterogeneous KV state cleanly. Verify the predicted savings on the real engine: one pinned version reported 90,058 bytes per token against a prediction of 90,112. section 12.7.2

How Rearchitecting LLMs covers it

How deep it goes
Explains preallocation and capacity effects, and notes uniform-page constraints, but leaves paging mechanics largely implicit.
Examples
Compares dynamic and preallocated cache strategies, then checks predicted cache savings after attention pruning.
What you can do after
Evaluate how weight or KV-cache quantization changes token capacity, and verify pruning savings on the target engine.
How it connects
Connects KV-cache capacity with GPU memory budgeting, quantization, pruning, and batch size.

Also explained in

Compare how 3 books explain PagedAttention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.