liveBook · Manning

Absorption trick

As explained in Build a DeepSeek Model (From Scratch)

The absorption trick is an algebraic attention optimization that pre-combines fixed query and key projection matrices so attention scores can be computed from a compressed latent representation.

The absorption trick makes Multi-Head Latent Attention (MLA) scores computable directly from a compressed latent KV cache. It pre-combines fixed query and key projection weights, avoiding the need to cache or reconstruct full keys for the score calculation. section 3.3.2 section 3.3

Start with Q = XWq and K = (XWdkv)Wuk, where Wdkv down-projects the input and Wuk up-projects the latent representation into keys. By associativity, QKᵀ = X(WqWukᵀ)(XWdkv)ᵀ. The product WqWukᵀ is fixed during inference, so it can be pre-combined. The historical term XWdkv is the latent KV cache, cKV. section 3.3.2

Figure 3.4 The mathematical rearrangement of the attention score calculation in MLA.

Follow the rearrangement from full query and key projections to a pre-combined projection and the latent cache. section 3.3.2

Pre-combining weights does not eliminate runtime projection work. It changes which representation is used: the current input produces an absorbed query, which is multiplied by the transpose of the updated latent cache to obtain content scores. section 3.3.2 section 3.14.4

Updating the cache for one token

Suppose the historical latent cache has shape (4, 4). A new token embedding of shape (1, 8) is multiplied by Wdkv of shape (8, 4), producing a (1, 4) latent vector. Appending it gives a (5, 4) cache; the absorbed content query then multiplies its transpose to produce content scores. section 3.14.4

Standard Rotary Positional Encoding (RoPE) obstructs this rearrangement: its position-dependent rotations intervene between the weights that must be combined. Decoupled RoPE separates a cache-efficient content path from a position path; their scores are added to form the final attention scores. section 3.13 section 3.18

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives the matrix rearrangement and traces a cache update, while clarifying that runtime projection work remains.
Examples
Traces a new token from an eight-dimensional embedding into a four-dimensional latent cache.
What you can do after
Evaluate how pre-combining projections avoids full-key reconstruction for content scores and why standard RoPE blocks the rearrangement.
How it connects
Connects Multi-Head Latent Attention, latent KV caching, Rotary Positional Encoding, and decoupled RoPE.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.