Key-value cache
A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
Build a DeepSeek Model (From Scratch)
A key-value cache stores previously computed attention keys and values, or their compressed representations, for reuse during autoregressive decoding.
- How it is used
- Speeds autoregressive inference by reusing past attention keys and values instead of recomputing them.
- How deep it goes
- Derives the decode loop and memory formula, benchmarks caching, and implements a compressed cache with positional offsets.
- Examples
- Uses a five-token matrix update and a timed GPT-2 generation comparison to illustrate reuse and speedup.
- What you can do after
- Build a cached attention path and evaluate its memory and inference-speed trade-offs.
Read how Build a DeepSeek Model (From Scratch) explains key-value cache →
Rearchitecting LLMs
A key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.
- How it is used
- Uses the cache to avoid recomputing attention states during autoregressive generation while managing dynamic GPU memory.
- How deep it goes
- Explains reuse, footprint arithmetic, quantization cycles, and pruning trade-offs, then checks an engine allocation against the estimate.
- Examples
- Compares 28 versus 16 attention layers and checks a Llama-3.2-3B vLLM allocation on a Colab T4.
- What you can do after
- Evaluate cache footprint and memory trade-offs using layer count, KV heads, precision, sequence length, and batch size.
Quantization and Fast Inference
Runtime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.
- How it is used
- Uses cached attention state to avoid recomputing prior keys and values during autoregressive generation.
- How deep it goes
- Derives FP16 cache memory from architecture and context length, then compares serving occupancy and queue depth.
- Examples
- Compares Llama-2-7B and Llama-3-8B cache sizes and an FP16 versus AWQ serving sweep.
- What you can do after
- Evaluate KV-cache memory requirements and serving headroom using architecture parameters, occupancy, and queue depth.
Read how Quantization and Fast Inference explains key-value cache →
CUDA for LLMs
A KV cache stores previously computed attention keys and values for reuse during sequence generation.
- How it is used
- Avoids repeated key and value projection work during autoregressive transformer decoding.
- How deep it goes
- Explains cache updates, tensor shapes, memory cost, and a generation loop, without detailing a cache-management kernel.
- Examples
- Uses a five-token phrase, a 100-position work comparison, tensor shapes, and a PyTorch generation loop.
- What you can do after
- Evaluate caching’s computation-memory tradeoff and measure generation throughput and time to first token.
Transformers in Action
Key-value caching is an inference optimization technique that stores previously computed attention key and value vectors for reuse during autoregressive generation.
- How it is used
- Reduces repeated attention computation during autoregressive inference.
- How deep it goes
- Explains per-token storage and reuse and discusses memory growth, without implementing the cache itself.
- Examples
- Benchmarks 256-token generation with Llama 3.2 3B Instruct, comparing cached and uncached latency.
- What you can do after
- Apply it via Hugging Face Transformers by toggling use_cache during generation and comparing latency.
Domain-Specific Small Language Models
Stored attention-related intermediate values used to avoid recomputing past context during autoregressive inference.
- How it is used
- Uses cached attention state to speed up autoregressive token generation without recomputing all previous tokens.
- How deep it goes
- Explains prefill and reuse, benchmarks generation time, and calculates per-token storage without detailing cache implementation internals.
- Examples
- Compares GPT-Neo generation times on a T4 GPU and sizes its cache for three batched sequences.
- What you can do after
- Operate/configure it through Transformers `generate(use_cache=True)` and account for cache memory when sizing inference batches.
Read how Domain-Specific Small Language Models explains key-value cache →
Retrieval Augmented Generation, The Foundational Ideas
A decoder cache stores key and value attention states from previously processed tokens for reuse during autoregressive generation.
- How it is used
- Reduces decoding memory or latency by reusing attention states instead of reprocessing prior tokens.
- How deep it goes
- Explains cache eviction strategies and their tradeoffs, but stops short of implementation code or decoder internals.
- Examples
- Uses H2O, SnapKV, RazorAttention, ManifoldKV, and precomputed RAPTOR summary-node states as concrete examples.
- What you can do after
- Operate/configure cache-reduction methods while weighing decoding memory, latency, retained information, and accuracy loss.
Read how Retrieval Augmented Generation, The Foundational Ideas explains key-value cache →
Vibe Engineering
A KV cache is a stored collection of transformer attention keys and values from previously processed tokens that accelerates subsequent generation.
- How it is used
- Frames local-inference capacity around context length as well as model size, especially when choosing workloads for local hardware.
- How deep it goes
- Explains cache contents, linear growth, approximate memory costs, and mitigations, but does not detail attention-cache implementation.
- Examples
- Uses Qwen 2.5 Coder 7B, a 14B model, tab completion, and 100,000-token coding contexts as memory comparisons.
- What you can do after
- Evaluate local inference memory by accounting for context-dependent cache size alongside fixed model-weight requirements.
Context Engineering
A key-value cache stores previously computed transformer attention keys and values for reuse during generation.
- How it is used
- Reuses previously computed attention state during generation and supports shared-prefix reuse across requests.
- How deep it goes
- Demonstrates cache creation and token-by-token reuse, then discusses memory growth and serving-engine optimizations.
- Examples
- Uses a shared context and query with GPT-2 to demonstrate cached generation.
- What you can do after
- Apply it via PyTorch and Hugging Face Transformers using past_key_values during generation.
Also covered in
- Build a Reasoning Model (From Scratch)
- Deep Learning with R, Third Edition
- Build a Text-to-Image Generator (from Scratch)
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- inferenceInference is the process of applying a trained model to input data to produce predictions or other outputs.
Related concepts
- keyhas part · A key is an identifier or value used to locate or access an associated entry or property in a mapping or object.
- prompt cachingused by · Prompt caching stores and reuses previously processed prompt content or computations to reduce repeated processing, cost, or latency.
- quadratic time complexityaddresses · Quadratic time complexity is an asymptotic running-time growth rate proportional to the square of input size, typically expressed as O(n²).
Questions about key-value cache
What is key-value cache?
A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
Which Manning books explain key-value cache?
Build a DeepSeek Model (From Scratch), Rearchitecting LLMs, Quantization and Fast Inference, CUDA for LLMs, Transformers in Action, Domain-Specific Small Language Models, Retrieval Augmented Generation, The Foundational Ideas, Vibe Engineering and 1 more.
What should I know before learning key-value cache?
Manning books that teach key-value cache usually introduce attention, throughput, transformer architecture and inference first.
What is key-value cache a kind of?
Key-value cache is a kind of memory optimization. Memory optimization improves memory use or access efficiency, often by reducing storage requirements through compact representations or efficient access strategies.
What is key-value cache part of?
Key-value cache is part of runtime memory. Runtime memory is the memory occupied by a loaded model's parameters, attention state, and temporary computation storage during execution.
What are the parts of key-value cache?
Key-value cache includes key.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








