liveBook · Manning

vLLM

As explained in Domain-Specific Small Language Models

A high-throughput model serving library for running and benchmarking language models with efficient attention and API compatibility.

vLLM is an open-source library for language-model inference and serving. It targets a constraint that general-purpose model servers can handle poorly: generating many outputs while each request’s key–value (KV) cache grows at an unpredictable rate. It supports both local, offline inference and online serving through an OpenAI-compatible HTTP API. section 11.1 section 11.1.1 section 11.1.2

Its central optimization is PagedAttention. Rather than reserving one contiguous KV-cache region for a request’s maximum possible length, vLLM stores cache blocks noncontiguously. A centralized scheduler and KV-cache manager allocate blocks as generation proceeds. This reduces wasted space and fragmentation, leaving more memory available for batching and improving serving throughput. section 11.1

Choose the interface according to where inference runs:

  • Offline: create an LLM instance, then call generate for text completion or chat for conversations. SamplingParams controls options such as temperature and top-p. Examples use models from the Hugging Face Hub, an NVIDIA GPU, and a quantized GGUF model. section 11.1.1
  • Online: start a model with the vllm serve command and call its HTTP endpoint with an OpenAI-compatible client. The server supports completions, chat completions, and embeddings, plus extensions including scoring and reranking. section 11.1.2
import torch
from vllm import LLM

model_id = "openai-community/gpt2"
device = "cuda" if torch.cuda.is_available() else "cpu"
vllm_model = LLM(model_id, device=device)

A minimal offline setup selects a device and creates an LLM instance. Initialization also loads model assets, prepares the KV cache, and warms up the model; a subsequent generate call produces a completion. section 11.1.1

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
  model="microsoft/Phi-3-mini-4k-instruct",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

An OpenAI Python client can call a locally served vLLM model by setting the server URL and API key, then requesting a chat completion. section 11.1.2

Treat fast inference as a starting point, not a deployment measurement. Benchmark the served endpoint under representative requests before exposing it to applications. vLLM’s benchmark scripts report measures including request and token throughput, latency, and time to first token; the serving example uses ShareGPT requests. section 11.1.2 section 11.5

How Domain-Specific Small Language Models covers it

How deep it goes
Explains KV-cache fragmentation and PagedAttention’s block allocation, then demonstrates inference, HTTP serving, and benchmarking without detailing implementation code.
Examples
Uses GPT-2 completion, Phi-3 chat and HTTP requests, a GGUF model, and a ShareGPT-based serving benchmark.
What it assumes
Assumes familiarity with Python, language-model inference, and serving hardware; introduces the LLM API through a basic completion example.
What you can do after
Operate/configure it for offline inference or an OpenAI-compatible server, then evaluate serving latency and throughput with benchmark scripts.
How it connects
Connects PagedAttention, KV caches, batching, quantization, OpenAI-compatible APIs, and serving benchmarks.

Also explained in

Compare how 2 books explain vLLM →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.