vLLM
vLLM is an open-source inference and serving engine for large language models, designed for high-throughput and memory-efficient execution.
Domain-Specific Small Language Models
A high-throughput model serving library for running and benchmarking language models with efficient attention and API compatibility.
- How it is used
- Serves language models offline or behind an OpenAI-compatible HTTP endpoint, with memory management designed for high-throughput generation.
- How deep it goes
- Explains KV-cache fragmentation and PagedAttention’s block allocation, then demonstrates inference, HTTP serving, and benchmarking without detailing implementation code.
- Examples
- Uses GPT-2 completion, Phi-3 chat and HTTP requests, a GGUF model, and a ShareGPT-based serving benchmark.
- What you can do after
- Operate/configure it for offline inference or an OpenAI-compatible server, then evaluate serving latency and throughput with benchmark scripts.
Read how Domain-Specific Small Language Models explains vLLM →
Rearchitecting LLMs
vLLM is an open-source library and inference engine for efficient large language model serving.
- How it is used
- Uses vLLM to optimize LLM inference memory and assess context capacity, batch size, and generation throughput.
- How deep it goes
- Implements cache and weight quantization experiments, measures token capacity and throughput, and discusses hardware trade-offs.
- Examples
- Tests FP8 cache and NF4 weight options, and compares a float16 Llama-3.2-3B baseline on a Colab T4.
- What you can do after
- Operate vLLM with quantization options and measure KV-token capacity and generation throughput for a configured GPU.
Transformers in Action
vLLM is an open-source serving library for efficient inference of large language models.
- How it is used
- Serves large language models with optimized latency and throughput for real-time or batched inference.
- How deep it goes
- Explains paged attention, batching, speculative decoding, and GPU kernels, then demonstrates deployment through an API.
- Examples
- Demonstrates serving Google Gemma 3 4B through a command-line server and OpenAI-compatible chat-completions client.
- What you can do after
- Operate/configure vLLM to serve a supported transformer model through an OpenAI-style endpoint.
Self-Hosting LLMs at Scale
vLLM is an open-source inference and serving engine for large language models, designed to serve many users efficiently.
- How it is used
- Uses vLLM as the serving engine for a self-hosted language model service under organizational control.
- How deep it goes
- Defines the engine and its main responsibilities, while deferring detailed deployment and tuning procedures.
- Examples
- Uses a fictional law firm’s pilot-to-production journey and a Kubernetes deployment diagram as operational examples.
- What you can do after
- Apply vLLM to serve a model, measure and tune it under load, and run it on Kubernetes.
Also covered in
What to know first
- key-value cacheA key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- memory usageMemory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
Related concepts
- inference enginekind of · An inference engine is software that derives outputs or conclusions by executing a trained machine-learning model or evaluating facts and rules.
- batch sizeimplements · Batch size is the number of items grouped for processing together in a single batch or computational operation.
- Kubernetesused by · Kubernetes is an open-source platform for orchestrating containerized workloads and automating their deployment, scaling, and management.
- memory paginguses · Memory paging is a memory-management technique that divides memory into fixed-size pages and maps or transfers them independently between memory and storage.
- PagedAttentionimplements · PagedAttention is an attention-memory technique that stores key-value cache blocks in independently managed, potentially noncontiguous memory pages.
Questions about vLLM
What is vLLM?
vLLM is an open-source inference and serving engine for large language models, designed for high-throughput and memory-efficient execution.
Which Manning books explain vLLM?
Domain-Specific Small Language Models, Rearchitecting LLMs, Transformers in Action and Self-Hosting LLMs at Scale.
What should I know before learning vLLM?
Manning books that teach vLLM usually introduce key-value cache, fine-tuning, language model and memory usage first.
What is vLLM a kind of?
vLLM is a kind of inference engine. An inference engine is software that derives outputs or conclusions by executing a trained machine-learning model or evaluating facts and rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



