liveBook · Manning

vLLM

As explained in Self-Hosting LLMs at Scale

vLLM is an open-source inference and serving engine for large language models, designed to serve many users efficiently.

When an organization needs to serve a language model to many users on systems it controls, vLLM provides an open-source inference and serving engine. An inference engine loads a trained model and answers requests efficiently. section 1.0 section 1.1.6

vLLM processes requests from multiple users to keep the GPU busy, manages temporary conversation memory, and provides an API compatible with common commercial services. Unlike vLLM’s multi-user focus, llama.cpp and tools built on it, such as Ollama and LM Studio, are popular for running a model on a single machine. section 1.1.6

When one machine is no longer enough, Kubernetes can manage vLLM across several connected machines. Reliable operation still requires attention to model memory, load time, machine failures, and slow responses. section 1.1.6

Figure 1.1 Kubernetes runs programs across several connected machines. Its control plane decides which machine runs each copy of vLLM, restarts copies that fail, and adds machines and copies as traffic grows.

See how Kubernetes places and restarts vLLM copies and expands capacity as traffic grows. section 1.1.6

How Self-Hosting LLMs at Scale covers it

How deep it goes
Defines the engine and its main responsibilities, while deferring detailed deployment and tuning procedures.
Examples
Uses a fictional law firm’s pilot-to-production journey and a Kubernetes deployment diagram as operational examples.
What it assumes
Assumes comfort with Python and the command line, but no prior language-model or GPU experience.
What you can do after
Apply vLLM to serve a model, measure and tune it under load, and run it on Kubernetes.
How it connects
Connects vLLM to open-weight models, inference engines, commercial-service-compatible APIs, and Kubernetes.

Also explained in

Compare how 4 books explain vLLM →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Hosting LLMs at Scale. The section links point to the parts of the book it draws on.