Key-value cache

As explained in Transformers in Action

Key-value caching is an inference optimization technique that stores previously computed attention key and value vectors for reuse during autoregressive generation.

An autoregressive decoder generates one token at a time. Each new token needs attention over earlier tokens, but recomputing their key and value vectors at every step wastes work. A key-value (KV) cache stores those vectors after they are computed so later steps can reuse them. section 9.3

At each generation step:

• Compute a query for the new token and use stored key and value pairs for earlier tokens. section 9.3
• Compute the new token’s key and value vectors and append them to the cache for subsequent steps. section 9.3
• Budget for a cache that grows with sequence length and model layers; long contexts and large batches can make memory use substantial. section 3.2 section 9.3

Measure the trade-off

In Hugging Face Transformers, call model.generate with use_cache=True and then use_cache=False while keeping the prompt and max_new_tokens fixed. A Llama 3.2 3B Instruct benchmark generating 256 tokens took about 8.5 seconds with caching versus 17.7 seconds without it. The speedup comes with additional memory use, so production deployment also calls for cache management. section 9.3 section 3.2

Figure 9.4 At each step, the query is computed for the new token, while past key and value vectors are cached and reused. This avoids recomputation of the entire attention context, dramatically reducing inference time.

The diagram tracks the newly computed query and the growing set of stored key-value pairs across generation steps. section 9.3

How Transformers in Action covers it

How deep it goes

Explains incremental reuse, memory growth, and a latency benchmark, but does not detail cache layout or invalidation procedures.

Examples

Compares cached and uncached generation of 256 tokens with Llama 3.2 3B Instruct using Hugging Face Transformers.

What it assumes

Introduces key and value vectors through token-by-token generation without requiring prior knowledge of KV caching.

What you can do after

Apply it via Hugging Face Transformers by toggling use_cache in generate and measuring generation latency.

How it connects

Connects autoregressive generation and decoder-only transformers to attention vectors and inference optimization.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• quadratic time complexityaddresses · Quadratic time complexity is an asymptotic running-time growth rate proportional to the square of input size, commonly denoted O(n²).
• grouped-query attentionaddressed by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
• valuehas part · In attention, a value is a vector representation of input content that is combined with other values according to attention weights to produce an output.
• quantizationused by · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
• keyhas part · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
• context windowconfigured by · A context window is the bounded sequence of tokens or other input information a model can access when processing or generating an output.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.