liveBook · Manning

vLLM

vLLM is an open-source inference and serving engine for large language models, designed for high-throughput and memory-efficient execution.

How 4 Manning books explain it

Domain-Specific Small Language Models

A high-throughput model serving library for running and benchmarking language models with efficient attention and API compatibility.

How it is used
Serves language models offline or behind an OpenAI-compatible HTTP endpoint, with memory management designed for high-throughput generation.
How deep it goes
Explains KV-cache fragmentation and PagedAttention’s block allocation, then demonstrates inference, HTTP serving, and benchmarking without detailing implementation code.
Examples
Uses GPT-2 completion, Phi-3 chat and HTTP requests, a GGUF model, and a ShareGPT-based serving benchmark.
What you can do after
Operate/configure it for offline inference or an OpenAI-compatible server, then evaluate serving latency and throughput with benchmark scripts.

Read how Domain-Specific Small Language Models explains vLLM →

Rearchitecting LLMs

vLLM is an open-source library and inference engine for efficient large language model serving.

How it is used
Uses vLLM to optimize LLM inference memory and assess context capacity, batch size, and generation throughput.
How deep it goes
Implements cache and weight quantization experiments, measures token capacity and throughput, and discusses hardware trade-offs.
Examples
Tests FP8 cache and NF4 weight options, and compares a float16 Llama-3.2-3B baseline on a Colab T4.
What you can do after
Operate vLLM with quantization options and measure KV-token capacity and generation throughput for a configured GPU.

Read how Rearchitecting LLMs explains vLLM →

Transformers in Action

vLLM is an open-source serving library for efficient inference of large language models.

How it is used
Serves large language models with optimized latency and throughput for real-time or batched inference.
How deep it goes
Explains paged attention, batching, speculative decoding, and GPU kernels, then demonstrates deployment through an API.
Examples
Demonstrates serving Google Gemma 3 4B through a command-line server and OpenAI-compatible chat-completions client.
What you can do after
Operate/configure vLLM to serve a supported transformer model through an OpenAI-style endpoint.

Read how Transformers in Action explains vLLM →

Self-Hosting LLMs at Scale

vLLM is an open-source inference and serving engine for large language models, designed to serve many users efficiently.

How it is used
Uses vLLM as the serving engine for a self-hosted language model service under organizational control.
How deep it goes
Defines the engine and its main responsibilities, while deferring detailed deployment and tuning procedures.
Examples
Uses a fictional law firm’s pilot-to-production journey and a Kubernetes deployment diagram as operational examples.
What you can do after
Apply vLLM to serve a model, measure and tune it under load, and run it on Kubernetes.

Read how Self-Hosting LLMs at Scale explains vLLM →

Also covered in

What to know first

Related concepts

Questions about vLLM

What is vLLM?

vLLM is an open-source inference and serving engine for large language models, designed for high-throughput and memory-efficient execution.

Which Manning books explain vLLM?

Domain-Specific Small Language Models, Rearchitecting LLMs, Transformers in Action and Self-Hosting LLMs at Scale.

What should I know before learning vLLM?

Manning books that teach vLLM usually introduce key-value cache, fine-tuning, language model and memory usage first.

What is vLLM a kind of?

vLLM is a kind of inference engine. An inference engine is software that derives outputs or conclusions by executing a trained machine-learning model or evaluating facts and rules.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.