Chapter 2: Solving the inference bottleneck with the key-value cache
Introduces the chapter’s roadmap from inefficient autoregressive inference through KV caching and its memory cost to MQA and GQA as context for DeepSeek’s later architecture.
After this chapter you will be able to
- KV cache inference scope. State that the KV cache is relevant only during a language model’s inference stage.
- Pretraining and inference distinction. Distinguish pretraining, which adjusts model weights by learning from a large dataset, from inference, which uses fixed weights to predict the next token and generate text. State that this chapter concerns inference after training is complete.
- Autoregressive generation loop. Trace how an initial prompt becomes a growing context as the model predicts a token, appends it, and repeats until an end-of-sequence token or generation limit stops the process.
- GPT-2 generation loop. Trace how the example starts with a prompt, predicts the next token from the current sequence, and appends that token before repeating. Explain why generating each token requires another full model pass in this demonstration.
- Query key value projections. Describe how the input embedding matrix is multiplied by three separate trainable weight matrices to produce queries, keys, and values, and explain each representation’s role.
- Causal attention weight calculation. Trace how queries and keys produce token-to-token scores and how scaling, a causal mask, and softmax turn those scores into usable attention weights.
- Value-weighted context vectors. Explain how attention weights and values produce a context matrix and what one row represents for a token in the example.
- Independent multi-head attention. Describe how multiple heads use separate smaller Q, K, and V projections and calculate their attention distributions and context matrices independently.
- Final projection to logits. Describe how the final context matrix becomes a logits matrix, what each row predicts, and what a logit means.
- Final-row prediction relevance. Explain why predicting the token after the complete input uses only the last row of the logits matrix. Describe how the last token’s context vector can support that prediction because self-attention has incorporated information from previous tokens.
- Intuition behind redundant computation. Explain why autoregressive inference appears to repeat unnecessary work: the model reprocesses a growing sequence to generate each token, but needs only the last token’s context vector for its next prediction.
- Repeated processing of prior tokens. Trace how each generated token is appended to the input and explain why the next inference step processes tokens the model has already processed.
- Repeated attention calculations. Explain why appending a token leaves the earlier query, key, and value rows unchanged, and why the earlier attention scores are therefore calculated again even though only the new token’s context vector is used.
- Uncached attention's quadratic cost. Explain why an uncached decode step requires O(n²) attention-score work per head per layer, and why generating later tokens becomes progressively slower as the context grows.
- Cached decoding complexity. Explain how reusing cached keys and values reduces attention-score work from O(n²) to O(n) for a single decode step, and distinguish that per-step improvement from the cost of an entire generation.
- KV caching principle. Explain that the KV cache stores results of past computations so the model does not recalculate them for tokens it has already seen, changing the described computation time from quadratic to linear.
- Newest token attention requirements. Trace why predicting the next token requires only the newest token’s context vector, and explain why producing it requires that token’s query plus the full key and value matrices.
- Key value cache workflow. Explain why past keys and values are cached rather than recomputed, and trace how a new token’s projections are used to extend the cache and calculate attention.
- Cached token generation loop. Trace how a decode step computes the new token’s query, key, and value, combines the new key and value with cached ones, and uses them to predict the next token.
- Prefill and decode phases. Distinguish the prompt-processing prefill phase from token-by-token decode, and explain why keys and values—but not old queries—are cached.
- Observed KV cache speedup. Describe how the section compares generation with and without KV caching, and state the observed result and its limits.
- KV cache memory trade-off. Explain why avoiding repeated key and value calculations requires storing their results in GPU memory, and why that storage can become a bottleneck for large models and long contexts.
- Per-step memory bandwidth demand. Describe why KV-cached generation can become memory-bandwidth bound: each generation step reads the stored keys and values for previous tokens from GPU main memory into the compute cores.
- KV cache size factors. Explain how to calculate KV cache memory from the number of layers, batch size, attention heads, head size, sequence length, and bytes per stored value, accounting for both keys and values.
- KV cache deployment limits. Explain why KV cache memory becomes a deployment bottleneck as models and context windows grow, and how it limits batch size and maximum context length on given hardware.
- Shared keys and values. Explain that Multi-Query Attention proposes having all attention heads share the same key and value matrices to address the KV-cache memory problem.
- Shared MQA key value cache. Describe how MQA keeps a distinct query projection for each head while sharing one key projection and one value projection across heads, so inference caches one set of keys and values rather than a set per head.
- MQA cache scaling. Explain why MQA reduces KV-cache size by a factor equal to the number of attention heads, and describe the expected inference benefit.
- MQA memory expressivity trade-off. Explain why sharing one set of keys and values across MQA heads saves memory but can weaken their ability to represent distinct, subtle relationships, even though the heads retain unique queries.
- Shared key and value projections. Describe how the MQA module differs from standard multi-head attention: it projects keys and values to one head’s dimension, then repeats them across query heads while leaving the core attention calculation unchanged.
- Cache size and repetition. Explain why MQA needs a smaller key-value cache than MHA, and distinguish the example’s physical repetition of keys and values from what must be cached.
- GQA expressivity memory compromise. Explain why GQA is presented as a middle ground between MHA and MQA: it aims for substantial memory savings without giving up as much of the multi-head design’s expressivity.
- Grouped key and value sharing. Explain how GQA groups attention heads and shares key and value projections within each group while keeping those projections distinct between groups. Describe why this saves cache memory compared with MHA and restores some diversity lost in MQA.
- GQA cache size scaling. Explain how the GQA KV-cache formula depends on the number of groups, and compare that dependence with MHA and MQA.
- Group count trade-off. Describe how choosing g between one and the total number of heads balances KV-cache savings against expressivity, using the 32-head, 8-group example.
- Grouped key value projections. Explain how the described GQA module differs from MQA: it uses num_groups key and value projections, and query heads in each group attend to the corresponding key/value group. Describe how changing num_groups moves the module from MQA-like to MHA-like behavior.
- Head sharing trade-off. Explain how MQA and GQA reduce KV cache memory relative to full multi-head attention, and what they give up in return.
- Limits of head reduction. Distinguish choosing a point on the memory–performance trade-off from the proposed alternative of compressing information within each head.
Section by section
Introduction
Introduces the chapter’s roadmap from inefficient autoregressive inference through KV caching and its memory cost to MQA and GQA as context for DeepSeek’s later architecture.
2.1 The LLM inference loop: Generating text one token at a time
Introduces the boundary that the KV cache is relevant only during LLM inference.
- KV cache inference scope: State that the KV cache is relevant only during a language model’s inference stage.
Why it matters: Keeping the cache tied to inference prevents confusion about when it is useful.
Key terms: KV cache, inference stage
2.1.1 Distinguishing pretraining from inference
Distinguishes the weight-changing learning phase from the fixed-weight inference phase that this chapter studies.
- Pretraining and inference distinction: Distinguish pretraining, which adjusts model weights by learning from a large dataset, from inference, which uses fixed weights to predict the next token and generate text. State that this chapter concerns inference after training is complete.
Why it matters: The chapter's generation and caching discussion assumes a trained model whose weights are fixed.
Key terms: pretraining, inference, parameters, weights, next token
2.1.2 Autoregressive process: Appending tokens to build context
Explains how an LLM generates text by predicting one token at a time and appending each token to the context for the next prediction.
- Autoregressive generation loop: Trace how an initial prompt becomes a growing context as the model predicts a token, appends it, and repeats until an end-of-sequence token or generation limit stops the process.
Why it matters: This growing-context loop is the basis for understanding why repeated inference work becomes a problem.
Key terms: autoregressive generation, context, end-of-sequence token
2.1.3 Visualizing autoregressive generation with GPT-2
A GPT-2 example shows how autoregressive generation repeatedly feeds the growing sequence through the model to produce one new token at a time.
- GPT-2 generation loop: Trace how the example starts with a prompt, predicts the next token from the current sequence, and appends that token before repeating. Explain why generating each token requires another full model pass in this demonstration.
Why it matters: This repeated full pass sets up the chapter’s question of whether every computation is necessary.
Key terms: autoregressive loop, initial prompt, input_ids, full computational pass
2.2 The core task: Predicting the next token
This section previews the transformer computations involved in predicting the next token and introduces a diagram that maps the upcoming walkthrough.
2.2.1 From input embeddings to context vectors: A mathematical walkthrough
The section traces how input embeddings become context vectors through single-head attention and then shows how multiple heads produce and combine distinct contextual views.
- Query key value projections: Describe how the input embedding matrix is multiplied by three separate trainable weight matrices to produce queries, keys, and values, and explain each representation’s role.
Why it matters: These three roles are the starting point for every attention calculation in the walkthrough. - Causal attention weight calculation: Trace how queries and keys produce token-to-token scores and how scaling, a causal mask, and softmax turn those scores into usable attention weights.
Why it matters: The mask explains why a token’s context cannot incorporate tokens that come after it. - Value-weighted context vectors: Explain how attention weights and values produce a context matrix and what one row represents for a token in the example.
Why it matters: This step turns relevance weights into the contextualized token representations used by later layers. - Independent multi-head attention: Describe how multiple heads use separate smaller Q, K, and V projections and calculate their attention distributions and context matrices independently.
Why it matters: Independent heads can provide different contextual views of the same sequence. - Head output combination: Explain how the separate head context matrices become one output and why the final projection does more than change its size.
Why it matters: Combining the heads makes their separate contextual views usable as one representation by subsequent layers.
Key terms: input embedding matrix X, Query (Q), Key (K), Value (V), Wq, Wk, attention scores, Q, Keys.T, causal mask
2.2.2 From context vectors to logits
This section traces how context vectors pass through the remaining transformer blocks and become vocabulary-wide logits at every token position.
- Per-token feed-forward processing: Explain how the feed-forward network processes each context vector independently, unlike attention, while preserving the context matrix’s shape.
Why it matters: This distinction explains what the FFN adds without confusing its role with attention. - Repeated transformer block flow: Trace the context matrix through the rest of a transformer block and into subsequent blocks, including the roles and placement of its two residual connections.
Why it matters: The output projection receives representations that have passed through every transformer block, not just one FFN. - Final projection to logits: Describe how the final context matrix becomes a logits matrix, what each row predicts, and what a logit means.
Why it matters: Understanding what each logits row represents sets up the next section’s question of which row matters for next-token prediction.
Key terms: Feed-Forward Network (FFN), context vector, attention mechanism, transformer block, layer normalization, residual connections, final layer normalization, output layer, logit, logits matrix
2.2.3 The key insight: Why only the last row matters
The section explains why next-token generation uses only the final position’s logits, how those logits select a token, and why recalculating earlier positions is wasteful.
- Final-row prediction relevance: Explain why predicting the token after the complete input uses only the last row of the logits matrix. Describe how the last token’s context vector can support that prediction because self-attention has incorporated information from previous tokens.
Why it matters: This identifies the unnecessary work that motivates the inference optimizations discussed next. - Next-token selection steps: Trace how the final logits vector becomes a generated token: select the last row, apply softmax to obtain vocabulary probabilities, and use argmax to choose the highest-probability token.
Why it matters: It connects the model’s numerical output to the single token generated at each step.
Key terms: last row, logits matrix, context vector, self-attention, final logits vector, softmax, probability distribution, argmax
2.3 The problem of redundant computations
This section introduces the inference bottleneck: the model repeatedly processes a growing sequence even though only the last token’s context vector is needed for the next prediction.
- Intuition behind redundant computation: Explain why autoregressive inference appears to repeat unnecessary work: the model reprocesses a growing sequence to generate each token, but needs only the last token’s context vector for its next prediction.
Why it matters: This intuition identifies the wasted work that motivates a more efficient inference loop.
Key terms: autoregressive loop, context vector, redundant computations
2.3.1 Intuition: Are we calculating the same thing over and over?
This section traces successive inference steps to show that generating one new token repeatedly reprocesses earlier tokens, creating growing time and memory costs.
- Repeated processing of prior tokens: Trace how each generated token is appended to the input and explain why the next inference step processes tokens the model has already processed.
Why it matters: Spotting the repeated work identifies the inference bottleneck that caching will address. - Cost of growing context: Explain why repeatedly processing a longer input makes inference increasingly costly, using the section’s cumulative-work analogy and its identification of the costly attention operations.
Why it matters: The growing cost explains why avoiding repeated calculations matters as sequences get longer.
Key terms: autoregressive loop, inference step, reprocessing, O(n²), QKᵀ, attention-weights-times-value product
2.3.2 A mathematical proof: Visualizing repeated calculations
The section compares two consecutive inference steps to show mathematically that recomputing attention over the full sequence repeats work that does not contribute to the next-token prediction.
- Repeated attention calculations: Explain why appending a token leaves the earlier query, key, and value rows unchanged, and why the earlier attention scores are therefore calculated again even though only the new token’s context vector is used.
Why it matters: This repeated work is the inference bottleneck that the following caching section addresses. - Attention calculation data flow: Trace how the four-token example produces queries, keys, values, attention scores, weights, and context vectors, including the role of causal masking.
Why it matters: Following the data flow makes it possible to locate exactly which calculations repeat at the next inference step.
Key terms: inference, query, key, value, attention scores, context vector, input embeddings, Wq, Wk, Wv
2.3.3 The performance effect: From quadratic to linear complexity
This section explains why recomputing attention during inference makes each decode step increasingly expensive, and how caching changes the attention-work scaling.
- Uncached attention's quadratic cost: Explain why an uncached decode step requires O(n²) attention-score work per head per layer, and why generating later tokens becomes progressively slower as the context grows.
Why it matters: It identifies the repeated work that makes long-context inference expensive. - Cached decoding complexity: Explain how reusing cached keys and values reduces attention-score work from O(n²) to O(n) for a single decode step, and distinguish that per-step improvement from the cost of an entire generation.
Why it matters: It shows why caching makes the growth in attention work more manageable for long contexts.
Key terms: attention scores matrix, autoregressive generation, O(n²), decode step, KV cache, cached keys and values, O(n), O(n³)
2.4 The solution: Caching for efficiency
This section introduces the KV cache as a way to avoid repeated computations during token generation.
- KV caching principle: Explain that the KV cache stores results of past computations so the model does not recalculate them for tokens it has already seen, changing the described computation time from quadratic to linear.
Why it matters: It explains why saving earlier work can make token generation more efficient.
Key terms: caching, key-value cache, KV cache, quadratic, linear computation time
2.4.1 What to cache? A step-by-step derivation
The section works backward from the newest token’s context vector to show why inference caches past keys and values while computing a fresh query, key, and value for each new token.
- Newest token attention requirements: Trace why predicting the next token requires only the newest token’s context vector, and explain why producing it requires that token’s query plus the full key and value matrices.
Why it matters: This identifies the smallest attention calculation needed at each generation step. - Key value cache workflow: Explain why past keys and values are cached rather than recomputed, and trace how a new token’s projections are used to extend the cache and calculate attention.
Why it matters: Reusing unchanged projections avoids repeating work on the entire growing sequence.
Key terms: context vector, query vector, Key matrix, Value matrix, attention weights, key-value (KV) cache, query, key, value, input embedding
2.4.2 The new inference loop with KV caching
This section explains how KV caching changes autoregressive inference so each new token reuses earlier keys and values instead of recomputing them.
- Cached token generation loop: Trace how a decode step computes the new token’s query, key, and value, combines the new key and value with cached ones, and uses them to predict the next token.
Why it matters: This is the inference workflow that eliminates repeated key and value projections for past tokens. - Per-layer and per-head caches: Explain where KV caches are maintained within the transformer and why they are distinct.
Why it matters: A KV cache is not one shared pair of matrices for the whole model. - Prefill and decode phases: Distinguish the prompt-processing prefill phase from token-by-token decode, and explain why keys and values—but not old queries—are cached.
Why it matters: The two phases explain why processing the initial prompt differs from generating subsequent tokens.
Key terms: KV cache, query, key, value, attention weights, context vector, per-layer, per-head, attention heads, prefill phase
2.4.3 Demonstrating the speedup of KV caching
This section compares GPT-2 generation with and without KV caching to show its speed benefit while noting its memory cost.
- Observed KV cache speedup: Describe how the section compares generation with and without KV caching, and state the observed result and its limits.
Why it matters: The timed comparison connects the cache’s theoretical efficiency benefit to faster text generation. - Cache memory trade-off: Explain the cost that accompanies the generation speedup from KV caching.
Why it matters: Faster generation does not mean caching is free of resource costs.
Key terms: KV cache, use_cache, speedup, Key and Value matrices, memory
2.5 The dark side of the KV cache: The memory cost
The section explains how KV caching speeds generation by trading repeated computation for GPU memory use and memory-bandwidth demand.
- KV cache memory trade-off: Explain why avoiding repeated key and value calculations requires storing their results in GPU memory, and why that storage can become a bottleneck for large models and long contexts.
Why it matters: A faster inference method can still be limited by the memory needed to use it. - Per-step memory bandwidth demand: Describe why KV-cached generation can become memory-bandwidth bound: each generation step reads the stored keys and values for previous tokens from GPU main memory into the compute cores.
Why it matters: Having enough memory to hold the cache does not guarantee fast generation.
Key terms: KV Cache, key and value matrices, memory footprint, GPU memory, memory-bandwidth bound, HBM, on-chip compute cores
2.5.1 The KV cache formula: Deconstructing the size
This section breaks down the KV cache memory formula and shows how model size and context length affect its cost.
- KV cache size factors: Explain how to calculate KV cache memory from the number of layers, batch size, attention heads, head size, sequence length, and bytes per stored value, accounting for both keys and values.
Why it matters: The formula lets you estimate inference memory rather than treating the cache as an unknown overhead. - Cache memory scaling: Describe why longer contexts and larger models increase KV cache memory, and illustrate the impact with the GPT-2 and GPT-3 examples.
Why it matters: The scaling explains why cache memory can become a practical constraint for larger models and longer contexts.
Key terms: KV cache, layers, batch size, heads, head size, sequence length, context length
2.5.2 The scaling problem in practice
This section shows how KV cache memory can limit batch size and context length, even though caching speeds up inference.
- KV cache deployment limits: Explain why KV cache memory becomes a deployment bottleneck as models and context windows grow, and how it limits batch size and maximum context length on given hardware.
Why it matters: The memory cost explains why faster cached inference does not automatically make large-scale deployment easy. - Hypothetical large-cache examples: Describe how the section's examples illustrate the scale of standard KV cache memory, while distinguishing the hypothetical DeepSeek-V3-scale calculation from the real model's cache.
Why it matters: These examples make the deployment constraint tangible without mistaking a hypothetical estimate for DeepSeek-V3's actual memory use.
Key terms: KV cache, batch, context length, memory overhead, standard multi-head attention, DeepSeek-V3, Multi-Head Latent Attention, FP16/BF16
2.6 The memory-first approach: Multi-Query Attention (MQA)
Introduces Multi-Query Attention as a way to address KV-cache memory use by sharing keys and values across attention heads.
- Shared keys and values: Explain that Multi-Query Attention proposes having all attention heads share the same key and value matrices to address the KV-cache memory problem.
Why it matters: This shared-key-and-value idea is the starting point for understanding MQA's approach to KV-cache memory.
Key terms: Multi-Query Attention (MQA), KV Cache, attention heads, key and value matrices
2.6.1 The core idea: Sharing a single key and value projection
This section contrasts multi-head attention with Multi-Query Attention and explains how shared keys and values reduce what inference must cache.
- Independent keys and values: Explain that in standard multi-head attention, each head has its own learned key and value projections, producing different representations and allowing multiple perspectives.
Why it matters: This baseline explains both the memory problem MQA addresses and the diversity its sharing changes. - Shared MQA key value cache: Describe how MQA keeps a distinct query projection for each head while sharing one key projection and one value projection across heads, so inference caches one set of keys and values rather than a set per head.
Why it matters: The shared cache is the architectural change that makes MQA a memory-first approach.
Key terms: multi-head attention (MHA), attention head, Wk, Wv, Multi-Query Attention (MQA), query projection, key projection, value projection, KV cache
2.6.2 The impact on the KV cache formula
This section shows how sharing keys and values in MQA removes attention-head count from KV-cache size, while qualifying the savings as architectural rather than drop-in.
- MQA cache scaling: Explain why MQA reduces KV-cache size by a factor equal to the number of attention heads, and describe the expected inference benefit.
Why it matters: It connects MQA’s shared keys and values to a concrete memory reduction and its expected inference benefit. - Illustrative savings and adoption: Distinguish the section’s hypothetical model-size examples from actual MQA deployments, and explain why adopting MQA is not an inference-time switch.
Why it matters: It keeps striking hypothetical savings from being mistaken for measured results or an easy retrofit.
Key terms: MHA, Multi-Query Attention, KV cache, attention heads, n, counterfactual illustrations, GPT-3, DeepSeek-V3, retraining, uptraining
2.6.3 The performance trade-off: Loss of expressivity
This section explains why MQA’s memory savings come at the cost of the head specialization that makes multi-head attention expressive.
- Independent attention head specialization: Explain how separate learned key and value weights let MHA heads specialize in different relationships, using the two interpretations of the brush sentence as an example.
Why it matters: Head specialization establishes what MQA risks giving up to save memory. - MQA memory expressivity trade-off: Explain why sharing one set of keys and values across MQA heads saves memory but can weaken their ability to represent distinct, subtle relationships, even though the heads retain unique queries.
Why it matters: This trade-off explains why reducing cache memory is not the only consideration in attention design.
Key terms: Multi-Head Attention (MHA), attention head, key and value vectors, specialize, Multi-Query Attention (MQA), shared key and value, unique queries, loss of expressivity, memory-first
2.6.4 Implementing an MQA layer from scratch
This section shows how an MQA layer shares one key and value projection across query heads and why that reduces the KV cache.
- Shared key and value projections: Describe how the MQA module differs from standard multi-head attention: it projects keys and values to one head’s dimension, then repeats them across query heads while leaving the core attention calculation unchanged.
Why it matters: This is the architectural change you need to recognize when reading or writing an MQA layer. - Cache size and repetition: Explain why MQA needs a smaller key-value cache than MHA, and distinguish the example’s physical repetition of keys and values from what must be cached.
Why it matters: Separating forward-pass copies from cached state makes the memory benefit of MQA clear.
Key terms: MultiQueryAttention, MultiHeadAttention, W_k, W_v, d_head, .repeat(), KV cache, .expand(), num_heads
2.7 The middle ground: Grouped-Query Attention (GQA)
Introduces Grouped-Query Attention as a tunable compromise between multi-head attention’s expressivity and multi-query attention’s memory efficiency.
- GQA expressivity memory compromise: Explain why GQA is presented as a middle ground between MHA and MQA: it aims for substantial memory savings without giving up as much of the multi-head design’s expressivity.
Why it matters: This trade-off is the reason to consider GQA rather than choosing only MHA or MQA.
Key terms: Grouped-Query Attention (GQA), MHA, MQA, expressivity, memory efficiency, tunable knob
2.7.1 The core idea: Sharing keys and values within groups
This section explains how Grouped-Query Attention shares keys and values within groups of heads to balance cache memory savings with head diversity.
- Grouped key and value sharing: Explain how GQA groups attention heads and shares key and value projections within each group while keeping those projections distinct between groups. Describe why this saves cache memory compared with MHA and restores some diversity lost in MQA.
Why it matters: Grouping explains how GQA trades some cache memory savings for more diversity than MQA.
Key terms: Grouped-Query Attention, GQA, attention heads, Wk, Wv, KV cache
2.7.2 The tunable knob: Balancing memory and performance
This section shows how the number of GQA groups controls KV-cache memory use and the trade-off with model expressivity.
- GQA cache size scaling: Explain how the GQA KV-cache formula depends on the number of groups, and compare that dependence with MHA and MQA.
Why it matters: The formula lets you estimate how changing the group count changes cache memory. - Group count trade-off: Describe how choosing g between one and the total number of heads balances KV-cache savings against expressivity, using the 32-head, 8-group example.
Why it matters: Choosing an intermediate group count can ease the cache bottleneck without taking the full expressivity trade-off of MQA.
Key terms: GQA, KV cache, g, MHA, MQA, tunable knob, number of groups, expressivity, query heads, key/value head
2.7.3 Implementing a GQA layer from scratch
This section describes how a GQA module extends MQA by grouping key/value projections and using the number of groups to tune the memory–performance trade-off.
- Grouped key value projections: Explain how the described GQA module differs from MQA: it uses num_groups key and value projections, and query heads in each group attend to the corresponding key/value group. Describe how changing num_groups moves the module from MQA-like to MHA-like behavior.
Why it matters: The group count is the setting that lets GQA balance cache memory savings against model performance.
Key terms: Grouped-Query Attention, num_groups, query heads, Key and Value projections, MQA, MHA
2.8 The performance vs. memory trade-off
This section compares the memory–expressivity trade-offs of MHA, MQA, and GQA, then introduces compressing information within heads as a different way to approach the KV cache bottleneck.
- Head sharing trade-off: Explain how MQA and GQA reduce KV cache memory relative to full multi-head attention, and what they give up in return.
Why it matters: It explains why a smaller KV cache does not automatically come without a performance cost. - Limits of head reduction: Distinguish choosing a point on the memory–performance trade-off from the proposed alternative of compressing information within each head.
Why it matters: It identifies the limitation that motivates the next approach without assuming its mechanics have already been explained.
Key terms: KV cache, Multi-Query Attention (MQA), Grouped-Query Attention (GQA), multi-head attention (MHA), key and value projections, expressive power, performance-vs-memory curve, Multi-Head Latent Attention (MLA), compress the key and value matrices
2.9 Summary
Where this is used later
- What happens when a new token arrives? (chapter 3) · Provides the baseline for examining what happens when a new token arrives in MLA inference.
- (chapter intro) (chapter 3) · The memory cost of the KV cache.
- The cost of intelligence (chapter 8) · Introduces the memory cost of the KV cache.
- The cost of decoupling: A trade-off between memory and computation (chapter 3) · The practical scaling problem caused by KV-cache memory.
- Quantifying the gains: The final cache memory comparison (chapter 3) · Provides the practical cache-scaling problem illustrated by the 400 GB estimate.
- The new KV cache formula: A 64x reduction (chapter 3) · Provides a standard-attention memory comparison for MLA's cache reduction.
- MLA: The best of both worlds (chapter 3) · The expressivity trade-off of sharing keys and values in MQA.
- The Payoff: An Empirical Head-to-Head Comparison (chapter 3) · The performance-versus-memory trade-off among attention mechanisms.
Concepts explained in this chapter
- key-value cacheA key-value cache stores previously computed attention keys and values, or their compressed representations, for reuse during autoregressive decoding.
- multi-head attentionMulti-head self-attention computes attention over an input sequence through multiple parallel attention heads, with each head having its own learned projections.
- context vectorA context vector is a numerical representation summarizing information relevant to a model's prediction at a particular sequence position.
- multi-query attentionMulti-query attention uses multiple query heads that share a smaller set of key and value projections to reduce attention-memory requirements.
- keyA key is an attention representation against which a query is compared to determine the relevance of a sequence position.
- valueA value is an attention representation containing the information contributed by a token when its key is selected.
- attention weightAn attention weight is a context-dependent coefficient that determines how strongly an input representation contributes to an attention output.
- queryA query vector is an attention representation that specifies what information a token seeks from other tokens.
- attention scoreAn attention score is a scalar measuring the relevance between a query and a key before normalization into attention weights.
- context matrixA context matrix contains attention outputs for all sequence positions, with each row formed by weighting value vectors.
- attentionStandard attention is an attention mechanism that computes interactions between query representations and key-value representations to combine relevant information.
- autoregressive modelAutoregressive generation is a text-generation process that predicts one token at a time and feeds each generated token back into the context for the next prediction.
- grouped-query attentionGrouped-query attention partitions query heads into groups whose members share key and value projections while different groups retain distinct projections.
- linear layerA linear layer is a neural-network layer that maps input representations to outputs through a learned affine transformation.
- softmaxSoftmax is a function that converts a vector of scores into nonnegative values summing to one.
- generationGeneration is the process of producing a sequence of tokens from a language model, typically by sampling or selecting successive outputs.
- attention headAn attention head is an independently parameterized attention submodule that computes relationship-specific attention within a multi-head attention mechanism.
- masked attentionA causal mask is an attention constraint that prevents each token from accessing later tokens in the sequence.
- logitA logit is an unnormalized real-valued score assigned to a possible output token before conversion to probabilities.
- transformer blockA transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.
- expressive powerRepresentational capacity is the ability of a model architecture to represent functions, patterns, or input-output relationships.
- feedforward networkA feed-forward neural network transforms its inputs through layers whose connections do not form recurrent cycles.
- matrix-matrix multiplicationMatrix multiplication is an algebraic operation that combines rows of one matrix with columns of another to produce a matrix.
- forward passA forward pass is the computation that propagates input data through a neural network to produce model outputs.
- language modelA neural language model that learns statistical patterns from large text corpora to generate and understand human language.
- parameterA model parameter is a learned numerical value that determines part of a machine-learning model's computation.
- embeddingAn embedding is a learned vector representation of a discrete input such as a token, including its semantic information.
- logits matrixA logits matrix is a matrix of routing scores assigning each token a score for each available expert.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes, with the probabilities summing to one.
- multi-head latent attentionMulti-head latent attention is an attention mechanism that represents key-value information in a compressed latent space to reduce memory use while retaining model quality.
- end-of-sequence tokenAn end-of-text token is a special token marking the boundary between documents or text sequences.
- FP32Float32, also called FP32, is a 32-bit floating-point numerical format that uses four bytes per value.
- bfloat16BF16 is a 16-bit floating-point numerical format with a wide exponent range commonly used for machine-learning computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
