Paged attention organizes the key-value (KV) cache into smaller blocks rather than one contiguous allocation. Managed like virtual-memory pages, blocks can be paged in and out of active memory as needed. section 9.3
This organization supports inference by:
- Reducing memory pressure and helping systems handle long sequences without keeping all cache memory active. section 9.3
- Supporting parallel service of batched requests, where cache management and latency become more important. section 9.3
Listing 9.2 Using Gemma 3 model through vLLM
!python -m vllm.entrypoints.api_server
--model google/gemma-3-4b
--port 8000
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-3-4b",
messages=[
{"role": "user", "content": "Explain paged attention in simple terms."}
],
max_tokens=128,
)
print(response.choices[0].message["content"])
See a vLLM serving example using Gemma 3; vLLM is identified as an inference engine that implements paged attention. section 9.3
