liveBook · Manning

Latent kv matrix

As explained in Build a DeepSeek Model (From Scratch)

A latent KV cache stores compressed key-value representations for previously processed tokens so attention can reuse historical context without storing full key and value states.

A latent KV matrix, cKV, stores a compressed representation of each processed token for reuse in attention. Multi-Head Latent Attention (MLA) caches this single matrix instead of separate full-dimensional key and value matrices, reducing the memory needed for historical context. section 3.1 section 3.2.2

For input embeddings X, a learned down-projector produces cKV = X × Wdkv. Conceptually, separate learned up-projectors recover keys and values: K = cKV × Wuk and V = cKV × Wuv. Each attention head can have its own up-projectors, giving it distinct key and value representations from the shared latent cache. Compression may nevertheless discard detail; preserving head-specific projections does not guarantee that all information survives. section 3.3.1 section 3.5.2

Appending a token

Suppose four cached tokens form a cKV matrix of shape (4, 4). A new embedding of shape (1, 8) multiplied by Wdkv of shape (8, 4) yields a latent vector of shape (1, 4). Append it to obtain a (5, 4) cache; the first dimension grows with sequence length, independently of the latent dimension. section 3.14.4 section 3.2.2

Figure 3.34 Updating the Latent KV Cache. The new token's embedding is down-projected to create a new latent vector, which is then appended to the existing cache.

Follow the down-projection and append operation for a new token. section 3.14.4

Full keys and values can be reconstructed for attention, but an efficient inference calculation can avoid reconstructing the full key matrix. Matrix associativity combines the fixed query and key up-projection weights: content scores become Xcurrent × (Wq × Wukᵀ) × cKVhistoryᵀ. The combined projection can be prepared once, although runtime projection work remains. On the value side, the value up-projector can similarly be combined with the output projection. section 3.3.2 section 3.3.3 section 3.4.3

Standard rotary positional encoding (RoPE) complicates this rearrangement: position-dependent rotations separate the weights that must be combined. Decoupled RoPE keeps a separate positional-key cache alongside the latent content cache. Thus cKV alone holds the content history, not the entire cache in this architecture. section 3.13 section 3.15

Cache memory trade-off

With 128 heads of dimension 128, standard multi-head attention stores 32,768 key and value dimensions per token. A latent dimension of 512 plus a 64-dimensional positional key uses 576, about 57 times fewer. At two bytes per element, total cache size is layers × batch size × sequence length × 576 × 2 bytes. section 3.15

In a matched-parameter TinyStories experiment, MLA and multi-head attention achieved lower validation loss than grouped-query and multi-query attention. MLA used 12 MB of KV cache versus 48 MB for multi-head attention in that comparison; these results are specific to the tested models. section 3.17

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives projection and absorption equations, traces cache updates, and quantifies memory with decoupled positional keys.
Examples
Traces a cache growing from shape (4, 4) to (5, 4) and compares attention variants on TinyStories.
What you can do after
Evaluate cache memory and understand how projection absorption and decoupled positional keys affect inference.
How it connects
Connects multi-head latent attention to KV caching, rotary positional encoding, multi-head attention, and grouped-query attention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.