vLLM
vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.
Domain-Specific Small Language Models
A high-throughput model serving library for running and benchmarking language models with efficient attention and API compatibility.
- How it is used
- Supports offline inference, online API serving, quantized-model execution, and serving-performance benchmarking.
- How deep it goes
- Explains PagedAttention, KV-cache fragmentation, batching, APIs, deployment, and benchmarks, but defers detailed implementation code.
- Examples
- Demonstrates GPT-2 and Phi-3 Mini generation, chat templates, GGUF loading, OpenAI-client requests, and ShareGPT benchmarking.
- What you can do after
- Operate/configure local and HTTP language-model inference, then measure throughput, latency, and time to first token.
Read how Domain-Specific Small Language Models explains vLLM →
Transformers in Action
vLLM is an open-source serving library for efficient inference of large language models.
- How it is used
- Serves large language models with optimized latency and throughput for real-time or batched inference.
- How deep it goes
- Explains paged attention, batching, speculative decoding, and GPU kernels, then demonstrates deployment through an API.
- Examples
- Demonstrates serving Google Gemma 3 4B through a command-line server and OpenAI-compatible chat-completions client.
- What you can do after
- Operate/configure vLLM to serve a supported transformer model through an OpenAI-style endpoint.
Related concepts
- paged attentionimplements · Paged attention is an attention-memory technique that partitions a key-value cache into blocks managed like virtual-memory pages instead of requiring one contiguous allocation.
- batch sizeimplements · Batch size is the number of items grouped together for processing or computation as a single batch.
- PagedAttentionuses · PagedAttention is an attention implementation that stores key-value cache blocks in dynamically allocated, potentially noncontiguous memory pages during inference.
- key-value cacheuses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.