Rotary Position Embedding (RoPE) injects position into attention by rotating query and key vectors by position-dependent angles. Unlike adding positional vectors to token embeddings, this leaves the initial embeddings untouched and preserves the magnitude of each rotated vector. Rotation still changes direction, so it does not leave the vector’s representation entirely unchanged. section 3.11.1 section 3.10.3
To rotate a query or key within one attention head:
- Group its dimensions into two-dimensional pairs. A four-dimensional vector [x1, x2, x3, x4] becomes (x1, x2) and (x3, x4). section 3.11.2
- Rotate each pair by an angle determined by the token’s position; different pairs use different angles. For a pair (x, y), rotation by θ produces (x cos θ − y sin θ, x sin θ + y cos θ). section 3.11.2
- Reassemble the rotated pairs into the position-encoded vector. Apply the same process to keys before calculating attention scores. section 3.11.2 section 3.12
Figure 3.25 Each pair of dimensions is treated as a 2D vector and rotated by a position-dependent angle θ.

Visualize each dimension pair as a two-dimensional vector rotated by a position-dependent angle. section 3.11.2
Position matters in the query–key interaction: their dot product determines an attention score. RoPE puts the positional transformation at that interaction rather than adding it to token embeddings before the Transformer block. section 3.10.3 section 3.11.1
Standard RoPE creates a problem for Multi-Head Latent Attention (MLA). MLA’s absorption trick combines position-agnostic weight matrices in advance and caches a compressed key–value representation. A position-dependent rotation between the matrices prevents that fixed combination; recomputing full rotated historical keys at every generation step would undermine caching efficiency. section 3.13
Decoupled RoPE separates the attention calculation into a content path and a position path. The content path uses MLA without positional information, retaining absorption; the position path uses separate projections with RoPE. Their scores are added to produce the final attention scores. section 3.14
Updating the content cache
For a new token, an embedding of shape (1, 8) is multiplied by a down-projector of shape (8, 4), producing a latent vector of shape (1, 4). Appending it to a (4, 4) latent KV cache gives a (5, 4) cache. An absorbed content query then scores against the updated cache while the position path runs in parallel. section 3.14.4




