A decoupled attention architecture splits attention into two parallel paths: a content path for semantic information and a position path for relative-position information. It combines their scores before applying the resulting attention weights to a content-only Value matrix. section 3.16
Let Q = [Qc, Qr] and K = [Kc, Kr]. Their dot product becomes Attention Scores = (Qc * Kcᵀ) + (Qr * Krᵀ). The content term remains compatible with the absorption trick and latent KV cache, while the positional term receives RoPE. section 3.14.2
The implementation separates responsibilities explicitly:
- The content path projects queries, compresses inputs into c_kv, reconstructs content keys and values, and caches the compressed representation. section 3.16
- The position path creates specialized keys and queries, applies RoPE, and caches the resulting positional key tensor. section 3.16
- The combined scores produce attention weights, which multiply the content-only values to form the context vector. section 3.14.2
Autoregressive update for a new token
When the next token is “bright,” the content path down-projects its embedding into a new latent vector, appends that vector to the latent KV cache, and computes content scores with the absorbed query. In parallel, the position path computes the position-based scores; the two results are then combined. section 3.14.4
Figure 3.30 The final attention score calculation. The scores are the sum of the content-based scores and the position-based scores.

Use the score diagram to trace how content-based and position-based attention scores are added before producing the final attention output. section 3.14.2
