liveBook · Manning

Chapter 2: Solving the inference bottleneck with the key-value cache

What you will learn in chapter 2 of Build a DeepSeek Model (From Scratch)

Introduces the chapter’s roadmap from inefficient autoregressive inference through KV caching and its memory cost to MQA and GQA as context for DeepSeek’s later architecture.

After this chapter you will be able to

Section by section

Introduction

Introduces the chapter’s roadmap from inefficient autoregressive inference through KV caching and its memory cost to MQA and GQA as context for DeepSeek’s later architecture.

2.1 The LLM inference loop: Generating text one token at a time

Introduces the boundary that the KV cache is relevant only during LLM inference.

Key terms: KV cache, inference stage

2.1.1 Distinguishing pretraining from inference

Distinguishes the weight-changing learning phase from the fixed-weight inference phase that this chapter studies.

Key terms: pretraining, inference, parameters, weights, next token

2.1.2 Autoregressive process: Appending tokens to build context

Explains how an LLM generates text by predicting one token at a time and appending each token to the context for the next prediction.

Key terms: autoregressive generation, context, end-of-sequence token

2.1.3 Visualizing autoregressive generation with GPT-2

A GPT-2 example shows how autoregressive generation repeatedly feeds the growing sequence through the model to produce one new token at a time.

Key terms: autoregressive loop, initial prompt, input_ids, full computational pass

2.2 The core task: Predicting the next token

This section previews the transformer computations involved in predicting the next token and introduces a diagram that maps the upcoming walkthrough.

2.2.1 From input embeddings to context vectors: A mathematical walkthrough

The section traces how input embeddings become context vectors through single-head attention and then shows how multiple heads produce and combine distinct contextual views.

Key terms: input embedding matrix X, Query (Q), Key (K), Value (V), Wq, Wk, attention scores, Q, Keys.T, causal mask

2.2.2 From context vectors to logits

This section traces how context vectors pass through the remaining transformer blocks and become vocabulary-wide logits at every token position.

Key terms: Feed-Forward Network (FFN), context vector, attention mechanism, transformer block, layer normalization, residual connections, final layer normalization, output layer, logit, logits matrix

2.2.3 The key insight: Why only the last row matters

The section explains why next-token generation uses only the final position’s logits, how those logits select a token, and why recalculating earlier positions is wasteful.

Key terms: last row, logits matrix, context vector, self-attention, final logits vector, softmax, probability distribution, argmax

2.3 The problem of redundant computations

This section introduces the inference bottleneck: the model repeatedly processes a growing sequence even though only the last token’s context vector is needed for the next prediction.

Key terms: autoregressive loop, context vector, redundant computations

2.3.1 Intuition: Are we calculating the same thing over and over?

This section traces successive inference steps to show that generating one new token repeatedly reprocesses earlier tokens, creating growing time and memory costs.

Key terms: autoregressive loop, inference step, reprocessing, O(n²), QKᵀ, attention-weights-times-value product

2.3.2 A mathematical proof: Visualizing repeated calculations

The section compares two consecutive inference steps to show mathematically that recomputing attention over the full sequence repeats work that does not contribute to the next-token prediction.

Key terms: inference, query, key, value, attention scores, context vector, input embeddings, Wq, Wk, Wv

2.3.3 The performance effect: From quadratic to linear complexity

This section explains why recomputing attention during inference makes each decode step increasingly expensive, and how caching changes the attention-work scaling.

Key terms: attention scores matrix, autoregressive generation, O(n²), decode step, KV cache, cached keys and values, O(n), O(n³)

2.4 The solution: Caching for efficiency

This section introduces the KV cache as a way to avoid repeated computations during token generation.

Key terms: caching, key-value cache, KV cache, quadratic, linear computation time

2.4.1 What to cache? A step-by-step derivation

The section works backward from the newest token’s context vector to show why inference caches past keys and values while computing a fresh query, key, and value for each new token.

Key terms: context vector, query vector, Key matrix, Value matrix, attention weights, key-value (KV) cache, query, key, value, input embedding

2.4.2 The new inference loop with KV caching

This section explains how KV caching changes autoregressive inference so each new token reuses earlier keys and values instead of recomputing them.

Key terms: KV cache, query, key, value, attention weights, context vector, per-layer, per-head, attention heads, prefill phase

2.4.3 Demonstrating the speedup of KV caching

This section compares GPT-2 generation with and without KV caching to show its speed benefit while noting its memory cost.

Key terms: KV cache, use_cache, speedup, Key and Value matrices, memory

2.5 The dark side of the KV cache: The memory cost

The section explains how KV caching speeds generation by trading repeated computation for GPU memory use and memory-bandwidth demand.

Key terms: KV Cache, key and value matrices, memory footprint, GPU memory, memory-bandwidth bound, HBM, on-chip compute cores

2.5.1 The KV cache formula: Deconstructing the size

This section breaks down the KV cache memory formula and shows how model size and context length affect its cost.

Key terms: KV cache, layers, batch size, heads, head size, sequence length, context length

2.5.2 The scaling problem in practice

This section shows how KV cache memory can limit batch size and context length, even though caching speeds up inference.

Key terms: KV cache, batch, context length, memory overhead, standard multi-head attention, DeepSeek-V3, Multi-Head Latent Attention, FP16/BF16

2.6 The memory-first approach: Multi-Query Attention (MQA)

Introduces Multi-Query Attention as a way to address KV-cache memory use by sharing keys and values across attention heads.

Key terms: Multi-Query Attention (MQA), KV Cache, attention heads, key and value matrices

2.6.1 The core idea: Sharing a single key and value projection

This section contrasts multi-head attention with Multi-Query Attention and explains how shared keys and values reduce what inference must cache.

Key terms: multi-head attention (MHA), attention head, Wk, Wv, Multi-Query Attention (MQA), query projection, key projection, value projection, KV cache

2.6.2 The impact on the KV cache formula

This section shows how sharing keys and values in MQA removes attention-head count from KV-cache size, while qualifying the savings as architectural rather than drop-in.

Key terms: MHA, Multi-Query Attention, KV cache, attention heads, n, counterfactual illustrations, GPT-3, DeepSeek-V3, retraining, uptraining

2.6.3 The performance trade-off: Loss of expressivity

This section explains why MQA’s memory savings come at the cost of the head specialization that makes multi-head attention expressive.

Key terms: Multi-Head Attention (MHA), attention head, key and value vectors, specialize, Multi-Query Attention (MQA), shared key and value, unique queries, loss of expressivity, memory-first

2.6.4 Implementing an MQA layer from scratch

This section shows how an MQA layer shares one key and value projection across query heads and why that reduces the KV cache.

Key terms: MultiQueryAttention, MultiHeadAttention, W_k, W_v, d_head, .repeat(), KV cache, .expand(), num_heads

2.7 The middle ground: Grouped-Query Attention (GQA)

Introduces Grouped-Query Attention as a tunable compromise between multi-head attention’s expressivity and multi-query attention’s memory efficiency.

Key terms: Grouped-Query Attention (GQA), MHA, MQA, expressivity, memory efficiency, tunable knob

2.7.1 The core idea: Sharing keys and values within groups

This section explains how Grouped-Query Attention shares keys and values within groups of heads to balance cache memory savings with head diversity.

Key terms: Grouped-Query Attention, GQA, attention heads, Wk, Wv, KV cache

2.7.2 The tunable knob: Balancing memory and performance

This section shows how the number of GQA groups controls KV-cache memory use and the trade-off with model expressivity.

Key terms: GQA, KV cache, g, MHA, MQA, tunable knob, number of groups, expressivity, query heads, key/value head

2.7.3 Implementing a GQA layer from scratch

This section describes how a GQA module extends MQA by grouping key/value projections and using the number of groups to tune the memory–performance trade-off.

Key terms: Grouped-Query Attention, num_groups, query heads, Key and Value projections, MQA, MHA

2.8 The performance vs. memory trade-off

This section compares the memory–expressivity trade-offs of MHA, MQA, and GQA, then introduces compressing information within heads as a different way to approach the KV cache bottleneck.

Key terms: KV cache, Multi-Query Attention (MQA), Grouped-Query Attention (GQA), multi-head attention (MHA), key and value projections, expressive power, performance-vs-memory curve, Multi-Head Latent Attention (MLA), compress the key and value matrices

2.9 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 1: Introduction to DeepSeek Chapter 3: The DeepSeek breakthrough: Multi-Head Latent Attention (MLA) →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.