liveBook · Manning

Chapter 3: The DeepSeek breakthrough: Multi-Head Latent Attention (MLA)

What you will learn in chapter 3 of Build a DeepSeek Model (From Scratch)

The chapter introduction frames MLA as a response to the KV-cache memory trade-off and explains why positional awareness remains necessary.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction frames MLA as a response to the KV-cache memory trade-off and explains why positional awareness remains necessary.

Key terms: KV Cache, Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Attention (MHA), Multi-Head Latent Attention (MLA), positional awareness, Rotary Positional Encoding (RoPE)

3.1 MLA: The best of both worlds

Introduces MLA as a way to reduce KV-cache memory without giving up distinct key and value representations for each attention head.

Key terms: Multi-Head Attention, Multi-Query Attention, KV cache, expressivity, Multi-Head Latent Attention, latent vector, cKV, compression, reconstruction, Key and Value

3.2 The MLA architecture: A visual walkthrough

Introduces MLA’s architectural shift: the input passes through a compressed latent space rather than being projected directly into keys and values.

Key terms: Multi-Head Latent Attention, compressed latent space, Keys and Values, projection layers

3.2.1 The query path (unchanged)

Explains how the simplified MLA walkthrough forms queries and clarifies how its query path differs from full DeepSeek-V3.

Key terms: query path, input embedding matrix X, Wq weight matrix, Query matrix Q, d_head, query latent, key/value compression, full DeepSeek-V3 architecture

3.2.2 The key/value path (the innovation)

This section shows how MLA compresses key/value information into one cached latent matrix and reconstructs separate keys and values when attention needs them.

Key terms: KV Down-Projector, Wdkv, Latent KV Matrix, cKV, latent space, Key Up-Projector, Wuk, Value Up-Projector, Wuv, Key matrix

3.3 The mathematical magic: How the latent matrix helps

This section introduces the claim that rearranging MLA’s attention calculation makes the latent matrix sufficient to cache past information while retaining unique attention heads.

Key terms: latent cKV matrix, absorption trick, cache, heads

3.3.1 A Step-by-Step Derivation of Q, K, and V in MLA

Derives how MLA forms queries directly from input embeddings while forming keys and values through a shared compressed latent matrix.

Key terms: input embedding matrix X, Query Matrix (Q), Latent KV Matrix (cKV), Key Matrix (K), Value Matrix (V)

3.3.2 The absorption trick: How attention scores are calculated

This section rearranges MLA’s attention-score calculation so it can use a fixed, pre-combined projection and the compact latent KV cache instead of a full Key matrix.

Key terms: absorption trick, Wq, Wuk, attention scores, latent KV matrix, cKV, KV cache, Key matrix

3.3.3 The final step: Calculating the context vector

This section shows how MLA reuses its cached latent matrix for the value side of attention by absorbing the value projection into the final output projection.

Key terms: cKV, Wuv, Wo, associative property of matrix multiplication, cKV cache, attention scores, context vector, unique, expressive projections

3.4 The new inference loop with MLA

This introduction frames MLA token generation as reusing cached latent representations rather than recalculating the preceding context.

Key terms: cached latent matrix, latent representations, cKV cache

3.4.1 What happens when a new token arrives?

This subsection explains the projections MLA performs when a new token arrives: a fresh query and a compressed latent vector for its key and value information.

Key terms: Query Vector, input embedding, KV Down-Projector (Wdkv), Latent Vector, Key and Value

3.4.2 Caching the latent vector: The only thing we store

This section explains how MLA updates its cache by appending a new token’s latent vector, keeping compressed history in one matrix.

Key terms: Latent Vector, cKV cache, latent matrix, Keys, Values

3.4.3 Decompressing the cache and calculating attention

This section shows how MLA temporarily reconstructs keys and values from its latent cache, then uses the current token’s query to calculate the context vector needed for next-token prediction.

Key terms: cKV Cache, Key Up-Projector (Wuk), Value Up-Projector (Wuv), Key Matrix K, Value Matrix V, Query Vector, Attention Weights, Context Vector

3.5 Quantifying the gains

Introduces the upcoming comparison of MLA’s cache-size reduction and performance preservation.

3.5.1 The new KV cache formula: A 64x reduction

This section compares MHA and MLA cache dimensions to quantify MLA’s memory reduction and qualify the headline 64× figure.

Key terms: MHA KV cache, MLA, d_latent, latent dimension, decoupled positional key, cached vector

3.5.2 Preserving performance: Why head diversity is maintained

This section explains how MLA keeps attention heads distinct despite sharing a compressed cache, while qualifying its performance claim.

Key terms: cKV, Wuk, Wuv, up-projection matrices, head diversity, MHA, latent compression, d_latent = 512, representational detail

3.6 Building an MLA module from scratch

This section flags a simplification in the MLA module and explains why attention still needs positional information.

Key terms: time-blind, word order, positional awareness, RMSNorm, latent representation, compression mechanism

3.7 The problem of order

This section explains why attention needs positional information and introduces positional encoding as a way to supply it without losing token meaning.

Key terms: attention mechanism, token embedding, context vector, sequence order, positional encoding, semantic embedding, positional information

3.8 Attempt #1: The naive approach - integer positional encodings

Introduces using a token’s position number as the simplest direct way to represent its position.

Key terms: position number, token's position

3.8.1 The simple idea: Using position numbers directly

This subsection shows how integer position numbers can distinguish identical tokens at different places in a sequence.

Key terms: Integer Positional Encoding, positional embedding, token embedding, input embeddings

3.8.2 The major flaw: Polluting semantic embeddings with large magnitudes

This subsection explains why adding large integer position values to token embeddings can drown out semantic information and what a better positional encoding must preserve.

Key terms: Integer Positional Encoding, token embeddings, positional embeddings, semantic information

3.9 Attempt #2: A step forward - Binary positional encodings

This section motivates binary positional encoding as a way to represent positions without letting large positional values overwhelm token embeddings.

Key terms: bounded encoding, Binary Positional Encoding, semantic token embeddings

3.9.1 Solving the magnitude problem with binary representation

This subsection shows how binary positional encodings avoid the magnitude problem of raw integer positions.

Key terms: binary representation, positional embedding vector, magnitude problem, token embedding

3.9.2 Uncovering a deeper pattern: Oscillation frequencies

The section shows that the bits in binary positional encodings change at different rates, providing both fine- and coarse-grained position information.

Key terms: binary positional encoding, Least Significant Bit (LSB), Most Significant Bit (MSB), oscillation frequency, fine-grained information, coarse-grained information

3.9.3 The new problem: The issue with discontinuous jumps

This section explains why binary positional encodings’ abrupt jumps are difficult for gradient-based training and motivates smooth sinusoidal signals.

Key terms: binary positional encoding, discrete jumps, square waves, gradient-based optimization, sinusoidal functions

3.10 Attempt #3: The "Attention Is All You Need" breakthrough - sinusoidal positional encodings

This section motivates sinusoidal positional encodings as a smooth alternative to integer and binary position encodings.

Key terms: Sinusoidal Positional Encodings, sine and cosine functions, multi-frequency oscillating patterns

3.10.1 From discrete jumps to smooth waves: Introducing sine and cosine

This subsection explains how sinusoidal positional encodings replace binary jumps with smooth sine and cosine waves while retaining signals at different frequencies.

Key terms: sinusoidal positional encodings, binary encoding, sin, cos, gradient-based optimization, frequency, index i, fine-grained local information, coarse-grained global information

3.10.2 The power of rotation: Encoding relative positions

This section explains how paired sine and cosine values turn differences between token positions into consistent rotations.

Key terms: sinusoidal positional encoding, dimension pair, 2D vector, relative positional relationships, rotation, ωᵢ, sin(A+B), cos(A+B)

3.10.3 The remaining flaw: Still polluting the token embeddings

The section explains why additive sinusoidal encodings can entangle position with token meaning and introduces the two ideas behind RoPE as a cleaner alternative.

Key terms: sinusoidal positional encodings, token embeddings, semantic pollution, representation entanglement, Query and Key vectors, attention scores, rotation, Rotary Positional Encoding (RoPE)

3.11 The state-of-the-art: Rotary Positional Encoding (RoPE)

Introduces RoPE as a response to the remaining drawback of sinusoidal positional encodings: adding them directly to semantic token embeddings.

Key terms: Rotary Positional Encoding (RoPE), sinusoidal encodings, relative positions, semantic token embeddings

3.11.1 The core insights: Injecting position into attention and preserving magnitude

This section explains why RoPE puts positional information in attention’s query and key vectors and uses rotation to preserve their magnitude.

Key terms: RoPE, Query and Key vectors, attention mechanism, token embeddings, Rotary Positional Encoding, rotation, magnitude

3.11.2 The mechanism: Rotating query and key vectors

Explains how RoPE rotates paired query and key dimensions to encode position while preserving vector magnitude and making relative distance available to attention.

Key terms: RoPE, Query, Key, 2D pairs, position-encoded vector, rotation angle θ, absolute position p, pair index i, multi-frequency oscillation, magnitude

3.12 The new challenge: Why standard RoPE and MLA don't mix

This section explains why applying standard RoPE to MLA’s cached keys creates a conflict between position-dependent rotation and a position-agnostic latent cache.

Key terms: Multi-Head Latent Attention (MLA), Rotary Positional Encoding (RoPE), position-agnostic latent representation, down-projector

3.13 The incompatibility problem: Why standard MLA and RoPE don’t work together

Explains why applying standard RoPE to MLA breaks the weight-matrix absorption that makes its cache efficient.

Key terms: absorption trick, RoPE, position-dependent, position-agnostic, latent matrix, Key matrix

3.14 The DeepSeek solution: Decoupled rotary position embedding

Introduces decoupled rotary position embedding as a way to combine MLA’s absorption trick with RoPE-based position information.

Key terms: Decoupled Rotary Position Embedding, Content Path, Position Path, absorption trick, RoPE

3.14.1 The new architecture: A visual and mathematical deep dive

This section separates MLA into a RoPE-free content path and a RoPE-encoded position path, showing what each produces and what must be cached.

Key terms: content path, cQ, cKV, Qc, Kc, Vc, position path, RoPE, Wkr, Kr

3.14.2 Combining the paths to calculate final attention

This section shows how decoupled content and positional paths combine to produce attention scores and a context vector while keeping content-path absorption compatible with RoPE.

Key terms: Qc, Qr, Kc, Kr, attention weights, content-only Value matrix, decoupled design, absorption trick, position-agnostic, RoPE-infused scores

3.14.3 The cost of decoupling: A trade-off between memory and computation

This section explains why decoupled RoPE’s modest compute and parameter costs are worthwhile when reading the KV cache is the main inference bottleneck.

Key terms: content scores, positional scores, FLOPs, Wqr, Wkr, memory bandwidth, KV cache, HBM, SRAM, long-context inference

3.14.4 The new inference loop in action: What happens when a new token arrives

This section traces how a newly generated token updates the content and position paths of decoupled MLA during inference.

Key terms: Content Path, absorption trick, Absorbed Content Query, Latent KV Cache, down-projector, Position Path, RoPE, Qr, Kr, Kr Cache

3.15 Quantifying the gains: The final cache memory comparison

This section derives the MLA with decoupled RoPE cache-size formula and uses DeepSeek-V2-scale dimensions to compare its memory footprint with standard MHA.

Key terms: Latent KV Cache, Decoupled Key Cache, d_latent, d_rope, MHA Cache Dimension, MLA+RoPE Cache Dimension

3.16 Building MLA + decoupled RoPE from scratch

This section maps decoupled MLA and RoPE into a PyTorch attention module with separate content and position paths that combine to produce a context vector.

Key terms: content path, position path, c_kv, q_c, k_c, v_c, forward method, attention scores, attention weights, content-only Value

3.17 The Payoff: An Empirical Head-to-Head Comparison

This section compares four attention mechanisms under similar training conditions and reports that MLA retains MHA-like learning performance while using a smaller KV cache.

Key terms: Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), learnable parameters, validation loss, validation perplexity, TinyStories, expressivity, KV Cache

3.18 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: Solving the inference bottleneck with the key-value cache Chapter 4: Mixture-of-Experts (MoE) in DeepSeek: Scaling intelligence efficiently →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.