vLLM is an open-source library for language-model inference and serving. It targets a constraint that general-purpose model servers can handle poorly: generating many outputs while each request’s key–value (KV) cache grows at an unpredictable rate. It supports both local, offline inference and online serving through an OpenAI-compatible HTTP API. section 11.1 section 11.1.1 section 11.1.2
Its central optimization is PagedAttention. Rather than reserving one contiguous KV-cache region for a request’s maximum possible length, vLLM stores cache blocks noncontiguously. A centralized scheduler and KV-cache manager allocate blocks as generation proceeds. This reduces wasted space and fragmentation, leaving more memory available for batching and improving serving throughput. section 11.1
Choose the interface according to where inference runs:
- Offline: create an LLM instance, then call generate for text completion or chat for conversations. SamplingParams controls options such as temperature and top-p. Examples use models from the Hugging Face Hub, an NVIDIA GPU, and a quantized GGUF model. section 11.1.1
- Online: start a model with the vllm serve command and call its HTTP endpoint with an OpenAI-compatible client. The server supports completions, chat completions, and embeddings, plus extensions including scoring and reranking. section 11.1.2
import torch
from vllm import LLM
model_id = "openai-community/gpt2"
device = "cuda" if torch.cuda.is_available() else "cpu"
vllm_model = LLM(model_id, device=device)
A minimal offline setup selects a device and creates an LLM instance. Initialization also loads model assets, prepares the KV cache, and warms up the model; a subsequent generate call produces a completion. section 11.1.1
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="token-abc123",
)
completion = client.chat.completions.create(
model="microsoft/Phi-3-mini-4k-instruct",
messages=[
{"role": "user", "content": "Hello!"}
]
)
An OpenAI Python client can call a locally served vLLM model by setting the server URL and API key, then requesting a chat completion. section 11.1.2
Treat fast inference as a starting point, not a deployment measurement. Benchmark the served endpoint under representative requests before exposing it to applications. vLLM’s benchmark scripts report measures including request and token throughput, latency, and time to first token; the serving example uses ShareGPT requests. section 11.1.2 section 11.5

