Multi-head latent attention (MLA) reduces the memory cost of the key-value (KV) cache by storing a compressed representation of each token’s keys and values. Unlike multi-query or grouped-query attention, which save memory by sharing key and value projections across heads, MLA compresses their information while retaining head-specific reconstructions. section 3.1 section 2.8
For input X, a down-projection produces the latent representation cKV = X · Wdkv. Separate learned up-projections reconstruct keys and values from cKV; head-specific projections give each head its own view. In a simplified implementation, queries are projected directly from X. Full DeepSeek-V3 also compresses queries, but does not cache the query latent. section 3.1 section 3.5.2 section 3.2.1
During inference, an absorption rearrangement can precompute the product of the query and key up-projection weights. This allows attention scores to be calculated using the smaller cached latent rather than caching full keys. Compression adds implementation complexity, and the possibility of losing fine-grained information remains an open question. section 3.13 section 3.5.2 section 1.2.1
Estimating cache savings
With 128 heads of dimension 128, a conventional key-value cache stores 2 × 128 × 128 = 32,768 values per token. A 512-dimensional MLA latent reduces that content-cache factor by 64×. A real DeepSeek cache also stores a decoupled positional key of about 64 dimensions, making the total 576 values per token and the reduction somewhat smaller. section 3.5.1
Standard rotary positional encoding (RoPE) complicates weight absorption: its position-dependent rotations prevent the query and key weights from being combined into one fixed matrix. Decoupled RoPE separates the computation into a content path and a positional path. The content path caches cKV; the position path applies RoPE and caches a rotated positional key. Their attention scores are added, and the resulting weights are applied to content values. section 3.13 section 3.16
Figure 3.36 A detailed flowchart of the DeepSeekAttention module. The input tensor x is processed through two parallel streams: the blue Content Path, which uses pure MLA for cache efficiency, and the purple Position Path, which applies RoPE. The two cached tensors, c_kv and k_r, are highlighted with key icons. The outputs are combined to produce the final context vector.

Trace the two paths, their cached tensors, and their combination into the context vector. section 3.16
A practical PyTorch implementation starts with linear down- and up-projections, reshapes queries, keys, and values into heads, and computes masked attention. The simplified core module leaves caching to the inference loop and omits latent RMS normalization; a complete design also needs positional handling and normalization. section 3.6 section 3.16
A controlled comparison
In a four-model experiment with comparable parameter counts and matched training settings, MLA achieved validation loss 2.3089 with a 12 MB KV cache, versus 2.3721 and 48 MB for multi-head attention. These are results for the tested models, not a guarantee for every workload. section 3.17
