liveBook · Manning

Decoupled attention architecture

As explained in Build a DeepSeek Model (From Scratch)

A decoupled attention architecture separates semantic-content processing from relative-position processing and fuses their attention scores.

A decoupled attention architecture splits attention into two parallel paths: a content path for semantic information and a position path for relative-position information. It combines their scores before applying the resulting attention weights to a content-only Value matrix. section 3.16

Let Q = [Qc, Qr] and K = [Kc, Kr]. Their dot product becomes Attention Scores = (Qc * Kcᵀ) + (Qr * Krᵀ). The content term remains compatible with the absorption trick and latent KV cache, while the positional term receives RoPE. section 3.14.2

The implementation separates responsibilities explicitly:

  • The content path projects queries, compresses inputs into c_kv, reconstructs content keys and values, and caches the compressed representation. section 3.16
  • The position path creates specialized keys and queries, applies RoPE, and caches the resulting positional key tensor. section 3.16
  • The combined scores produce attention weights, which multiply the content-only values to form the context vector. section 3.14.2

Autoregressive update for a new token

When the next token is “bright,” the content path down-projects its embedding into a new latent vector, appends that vector to the latent KV cache, and computes content scores with the absorbed query. In parallel, the position path computes the position-based scores; the two results are then combined. section 3.14.4

Figure 3.30 The final attention score calculation. The scores are the sum of the content-based scores and the position-based scores.

Use the score diagram to trace how content-based and position-based attention scores are added before producing the final attention output. section 3.14.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Descends from score identity to parallel inference steps and executable PyTorch module components, including cache updates.
Examples
Uses the token “bright,” matrix-shaped cache growth, score equations, flowcharts, and staged PyTorch listings.
What it assumes
Assumes the absorption trick, latent KV caching, MLA projections, and RoPE are understood well enough to combine.
What you can do after
Build it from scratch as a PyTorch DeepSeekAttention module with content and RoPE position paths.
How it connects
Connects multi-head latent attention, RoPE, the absorption trick, latent KV caching, attention weights, and context vectors.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.