vLLM is an open-source inference engine for serving large language models efficiently. Its architecture combines paged attention, continuous batching, speculative decoding, and optimized GPU operations to improve latency and throughput without modifying model weights. section 9.3
vLLM is useful when deployment must:
- Serve large models without requiring all memory to remain allocated contiguously. section 9.3
- Batch multiple requests efficiently while maintaining low serving latency. section 9.3
- Run models already reduced through pruning, quantization, or distillation more efficiently. section 9.3
Serve Gemma 3 through an OpenAI-style API
Start vLLM with `python -m vllm.entrypoints.api_server --model google/gemma-3-4b --port 8000`, then send chat completions to `http://localhost:8000/v1` using an OpenAI-compatible client. section 9.3
Listing 9.2 Using Gemma 3 model through vLLM
!python -m vllm.entrypoints.api_server
--model google/gemma-3-4b
--port 8000
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-3-4b",
messages=[
{"role": "user", "content": "Explain paged attention in simple terms."}
],
max_tokens=128,
)
print(response.choices[0].message["content"])
Shows the command-line server configuration and client request used to query Gemma 3 through vLLM. section 9.3
