liveBook · Manning

Multi-head latent attention

As explained in Build a DeepSeek Model (From Scratch)

Multi-head latent attention is an attention mechanism that represents key-value information in a compressed latent space to reduce memory use while retaining model quality.

Multi-head latent attention (MLA) reduces the memory cost of the key-value (KV) cache by storing a compressed representation of each token’s keys and values. Unlike multi-query or grouped-query attention, which save memory by sharing key and value projections across heads, MLA compresses their information while retaining head-specific reconstructions. section 3.1 section 2.8

For input X, a down-projection produces the latent representation cKV = X · Wdkv. Separate learned up-projections reconstruct keys and values from cKV; head-specific projections give each head its own view. In a simplified implementation, queries are projected directly from X. Full DeepSeek-V3 also compresses queries, but does not cache the query latent. section 3.1 section 3.5.2 section 3.2.1

During inference, an absorption rearrangement can precompute the product of the query and key up-projection weights. This allows attention scores to be calculated using the smaller cached latent rather than caching full keys. Compression adds implementation complexity, and the possibility of losing fine-grained information remains an open question. section 3.13 section 3.5.2 section 1.2.1

Estimating cache savings

With 128 heads of dimension 128, a conventional key-value cache stores 2 × 128 × 128 = 32,768 values per token. A 512-dimensional MLA latent reduces that content-cache factor by 64×. A real DeepSeek cache also stores a decoupled positional key of about 64 dimensions, making the total 576 values per token and the reduction somewhat smaller. section 3.5.1

Standard rotary positional encoding (RoPE) complicates weight absorption: its position-dependent rotations prevent the query and key weights from being combined into one fixed matrix. Decoupled RoPE separates the computation into a content path and a positional path. The content path caches cKV; the position path applies RoPE and caches a rotated positional key. Their attention scores are added, and the resulting weights are applied to content values. section 3.13 section 3.16

Figure 3.36 A detailed flowchart of the DeepSeekAttention module. The input tensor x is processed through two parallel streams: the blue Content Path, which uses pure MLA for cache efficiency, and the purple Position Path, which applies RoPE. The two cached tensors, c_kv and k_r, are highlighted with key icons. The outputs are combined to produce the final context vector.

Trace the two paths, their cached tensors, and their combination into the context vector. section 3.16

A practical PyTorch implementation starts with linear down- and up-projections, reshapes queries, keys, and values into heads, and computes masked attention. The simplified core module leaves caching to the inference loop and omits latent RMS normalization; a complete design also needs positional handling and normalization. section 3.6 section 3.16

A controlled comparison

In a four-model experiment with comparable parameter counts and matched training settings, MLA achieved validation loss 2.3089 with a 12 MB KV cache, versus 2.3721 and 48 MB for multi-head attention. These are results for the tested models, not a guarantee for every workload. section 3.17

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops latent projections, weight absorption, decoupled RoPE, and PyTorch implementations, while distinguishing simplified components from the full design.
Examples
Uses a 128-head cache-size calculation and a controlled four-model comparison of validation loss and cache memory.
What it assumes
Builds from standard attention and KV-cache fundamentals before introducing latent compression and positional handling.
What you can do after
Builds MLA from scratch in PyTorch and combines its content path with a decoupled positional path.
How it connects
Connects KV caching, multi-head attention, grouped-query attention, rotary positional encoding, and RMS normalization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.