liveBook · Manning

Key-value cache

Also called KV cache

A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.

How 9 Manning books explain it

Build a DeepSeek Model (From Scratch)

A key-value cache stores previously computed attention keys and values, or their compressed representations, for reuse during autoregressive decoding.

How it is used
Speeds autoregressive inference by reusing past attention keys and values instead of recomputing them.
How deep it goes
Derives the decode loop and memory formula, benchmarks caching, and implements a compressed cache with positional offsets.
Examples
Uses a five-token matrix update and a timed GPT-2 generation comparison to illustrate reuse and speedup.
What you can do after
Build a cached attention path and evaluate its memory and inference-speed trade-offs.

Read how Build a DeepSeek Model (From Scratch) explains key-value cache →

Rearchitecting LLMs

A key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.

How it is used
Uses the cache to avoid recomputing attention states during autoregressive generation while managing dynamic GPU memory.
How deep it goes
Explains reuse, footprint arithmetic, quantization cycles, and pruning trade-offs, then checks an engine allocation against the estimate.
Examples
Compares 28 versus 16 attention layers and checks a Llama-3.2-3B vLLM allocation on a Colab T4.
What you can do after
Evaluate cache footprint and memory trade-offs using layer count, KV heads, precision, sequence length, and batch size.

Read how Rearchitecting LLMs explains key-value cache →

Quantization and Fast Inference

Runtime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.

How it is used
Uses cached attention state to avoid recomputing prior keys and values during autoregressive generation.
How deep it goes
Derives FP16 cache memory from architecture and context length, then compares serving occupancy and queue depth.
Examples
Compares Llama-2-7B and Llama-3-8B cache sizes and an FP16 versus AWQ serving sweep.
What you can do after
Evaluate KV-cache memory requirements and serving headroom using architecture parameters, occupancy, and queue depth.

Read how Quantization and Fast Inference explains key-value cache →

CUDA for LLMs

A KV cache stores previously computed attention keys and values for reuse during sequence generation.

How it is used
Avoids repeated key and value projection work during autoregressive transformer decoding.
How deep it goes
Explains cache updates, tensor shapes, memory cost, and a generation loop, without detailing a cache-management kernel.
Examples
Uses a five-token phrase, a 100-position work comparison, tensor shapes, and a PyTorch generation loop.
What you can do after
Evaluate caching’s computation-memory tradeoff and measure generation throughput and time to first token.

Read how CUDA for LLMs explains key-value cache →

Transformers in Action

Key-value caching is an inference optimization technique that stores previously computed attention key and value vectors for reuse during autoregressive generation.

How it is used
Reduces repeated attention computation during autoregressive inference.
How deep it goes
Explains per-token storage and reuse and discusses memory growth, without implementing the cache itself.
Examples
Benchmarks 256-token generation with Llama 3.2 3B Instruct, comparing cached and uncached latency.
What you can do after
Apply it via Hugging Face Transformers by toggling use_cache during generation and comparing latency.

Read how Transformers in Action explains key-value cache →

Domain-Specific Small Language Models

Stored attention-related intermediate values used to avoid recomputing past context during autoregressive inference.

How it is used
Uses cached attention state to speed up autoregressive token generation without recomputing all previous tokens.
How deep it goes
Explains prefill and reuse, benchmarks generation time, and calculates per-token storage without detailing cache implementation internals.
Examples
Compares GPT-Neo generation times on a T4 GPU and sizes its cache for three batched sequences.
What you can do after
Operate/configure it through Transformers `generate(use_cache=True)` and account for cache memory when sizing inference batches.

Read how Domain-Specific Small Language Models explains key-value cache →

Retrieval Augmented Generation, The Foundational Ideas

A decoder cache stores key and value attention states from previously processed tokens for reuse during autoregressive generation.

How it is used
Reduces decoding memory or latency by reusing attention states instead of reprocessing prior tokens.
How deep it goes
Explains cache eviction strategies and their tradeoffs, but stops short of implementation code or decoder internals.
Examples
Uses H2O, SnapKV, RazorAttention, ManifoldKV, and precomputed RAPTOR summary-node states as concrete examples.
What you can do after
Operate/configure cache-reduction methods while weighing decoding memory, latency, retained information, and accuracy loss.

Read how Retrieval Augmented Generation, The Foundational Ideas explains key-value cache →

Vibe Engineering

A KV cache is a stored collection of transformer attention keys and values from previously processed tokens that accelerates subsequent generation.

How it is used
Frames local-inference capacity around context length as well as model size, especially when choosing workloads for local hardware.
How deep it goes
Explains cache contents, linear growth, approximate memory costs, and mitigations, but does not detail attention-cache implementation.
Examples
Uses Qwen 2.5 Coder 7B, a 14B model, tab completion, and 100,000-token coding contexts as memory comparisons.
What you can do after
Evaluate local inference memory by accounting for context-dependent cache size alongside fixed model-weight requirements.

Read how Vibe Engineering explains key-value cache →

Context Engineering

A key-value cache stores previously computed transformer attention keys and values for reuse during generation.

How it is used
Reuses previously computed attention state during generation and supports shared-prefix reuse across requests.
How deep it goes
Demonstrates cache creation and token-by-token reuse, then discusses memory growth and serving-engine optimizations.
Examples
Uses a shared context and query with GPT-2 to demonstrate cached generation.
What you can do after
Apply it via PyTorch and Hugging Face Transformers using past_key_values during generation.

Read how Context Engineering explains key-value cache →

Also covered in

What to know first

Related concepts

Questions about key-value cache

What is key-value cache?

A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.

Which Manning books explain key-value cache?

Build a DeepSeek Model (From Scratch), Rearchitecting LLMs, Quantization and Fast Inference, CUDA for LLMs, Transformers in Action, Domain-Specific Small Language Models, Retrieval Augmented Generation, The Foundational Ideas, Vibe Engineering and 1 more.

What should I know before learning key-value cache?

Manning books that teach key-value cache usually introduce attention, throughput, transformer architecture and inference first.

What is key-value cache a kind of?

Key-value cache is a kind of memory optimization. Memory optimization improves memory use or access efficiency, often by reducing storage requirements through compact representations or efficient access strategies.

What is key-value cache part of?

Key-value cache is part of runtime memory. Runtime memory is the memory occupied by a loaded model's parameters, attention state, and temporary computation storage during execution.

What are the parts of key-value cache?

Key-value cache includes key.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.