Inference optimization

As explained in Transformers in Action

Inference optimization is the improvement of a trained model's serving efficiency by reducing latency, increasing throughput, or lowering resource use without changing the model architecture.

Inference optimization improves serving efficiency without changing model architecture. It reduces latency, increases throughput, and lowers resource demands by changing operation scheduling, memory access, and GPU utilization during deployment. section 9.3

Autoregressive generation produces one token at a time, so recomputing the complete attention context at every step wastes work. Key-value (KV) caching stores each token’s key and value vectors, while the query is recomputed for the new token; later steps reuse the stored context. section 9.3

Common inference-optimization mechanisms include:

• KV caching avoids repeated attention computation, but its memory footprint grows with sequence length and batch size. section 9.3
• Paged attention manages attention memory efficiently during deployment. section 9.0 section 9.6
• FlashAttention reduces attention-layer memory and compute overhead while retaining exactness. section 9.6
• Operator fusion, KV caching, and paged attention improve deployment throughput and latency. section 9.6

Benchmarking cached generation

A Hugging Face Transformers benchmark loads meta-llama/Llama-3.2-3B-Instruct, generates up to 256 tokens with deterministic decoding, and compares runs with use_cache enabled and disabled. The comparison measures the latency effect of KV caching. section 9.3

Serving through vLLM

A vLLM API server serves google/gemma-3-4b on port 8000, after which an OpenAI-compatible client requests an explanation of paged attention. section 9.3

Figure 9.4 At each step, the query is computed for the new token, while past key and value vectors are cached and reused. This avoids recomputation of the entire attention context, dramatically reducing inference time.

The figure traces autoregressive attention step by step: each new token adds key-value pairs to the cache, while subsequent generation reuses them instead of recomputing the full context. section 9.3

How Transformers in Action covers it

How deep it goes

Explains scheduling, memory access, GPU use, KV-cache internals, and deployment mechanisms, while deferring implementation detail for some techniques.

Examples

Uses a Llama 3.2 latency benchmark and a Gemma 3 service accessed through vLLM and an OpenAI-compatible client.

What it assumes

Assumes familiarity with autoregressive transformers and attention, while introducing KV caching and deployment efficiency through concrete mechanics.

What you can do after

Apply it via Hugging Face Transformers and vLLM to benchmark caching and serve optimized transformer inference.

How it connects

Connects autoregressive transformers with KV caching, paged attention, FlashAttention, operator fusion, pruning, distillation, and sharding.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• key-value cacheuses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
• meanaddresses · A mean is a measure of central tendency, calculated as the arithmetic average of values or, for a random variable, as the probability-weighted average of its possible values.
• tradeoffuses · A tradeoff is a situation in which improving one objective or property requires accepting a cost, limitation, or reduction in another.
• specificationpart of · A specification is a formal or informal description of the required or intended behavior, properties, interfaces, or constraints of a system or component.
• quantizationhas part · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
• batchinghas part · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.