A position path separates positional relevance from semantic relevance when standard Rotary Positional Encoding (RoPE) conflicts with Multi-Head Latent Attention (MLA). The content path keeps MLA position-agnostic so its absorption trick remains valid; the position path derives RoPE-encoded queries and keys. Their attention scores are added before applying the weights to content-only values. section 3.14 section 3.18
Implement the position path as a parallel stream:
- Project the input X through a dedicated key matrix W_k_pos, apply RoPE, and share the resulting positional key projection across heads when possible. section 3.14.1
- Reuse the latent query cQ from the content path, project it through W_q_pos, reshape it into heads, and apply RoPE to obtain positional queries. section 3.14.1
- Add position-based scores to content-based scores, then use the combined attention weights with the content-only value matrix. section 3.16
Figure 3.29 The Position Path of the Decoupled Architecture. This new path creates specialized Query (Qr) and Key (Kr) matrices that are infused with Rotary Positional Encodings.

Use the architecture figure to track how RoPE-encoded Qr and Kr form a positional stream alongside content-based attention. section 3.14.1
A from-scratch PyTorch implementation groups separate linear layers under Content Path and Position Path headings. The forward pass reshapes each projection into attention heads, rotates only positional tensors, and retains the latent content tensor c_kv plus rotated key k_r for caching. section 3.16
Autoregressive update
When the next token is "bright," append its down-projected latent vector to the latent KV cache while computing its positional query and key through the RoPE path. Combine the new content and position scores to predict the following token. section 3.14.4
