vLLM is a high-throughput library for language-model inference and serving. It targets LLM-specific memory and batching problems rather than general machine-learning model serving. Its PagedAttention system stores key–value cache blocks noncontiguously, reducing fragmentation and enabling more flexible batching. section 11.1 section 11.5
Use vLLM in two main ways:
- Run models offline through the Python API, including text generation, chat, classification, scoring, rewards, and embeddings. section 11.1.1
- Serve models online through an OpenAI-compatible HTTP API supporting completions, chat completions, embeddings, tokenization, scoring, pooling, and reranking. section 11.1.2
- Configure generation with sampling parameters such as temperature, top_p, and top_k, and process multiple conversations in batches. section 11.1.1
- Run quantized models, including GGUF models, and deploy with Docker, Kubernetes, Nginx, or AWS. section 11.1.2 section 11.5
Offline generation and online benchmarking
A typical offline workflow installs vLLM, loads GPT-2 or Phi-3 Mini with the LLM class, configures SamplingParams, and calls generate or chat. An online workflow starts `vllm serve`, calls the endpoint with the OpenAI Python client, then benchmarks ShareGPT traffic for throughput, latency, and time to first token. section 11.1.1 section 11.1.2
import torch
from vllm import LLM
model_id = "openai-community/gpt2"
device = "cuda" if torch.cuda.is_available() else "cpu"
vllm_model = LLM(model_id, device=device)
Shows the minimal Python setup for loading a model with device selection before running inference. section 11.1.1
