liveBook · Manning

Rotary Position Embedding

As explained in Build a DeepSeek Model (From Scratch)

Rotary positional encoding is a positional-encoding technique that applies position-dependent rotations to query and key representations while preserving their magnitudes.

Rotary Position Embedding (RoPE) injects position into attention by rotating query and key vectors by position-dependent angles. Unlike adding positional vectors to token embeddings, this leaves the initial embeddings untouched and preserves the magnitude of each rotated vector. Rotation still changes direction, so it does not leave the vector’s representation entirely unchanged. section 3.11.1 section 3.10.3

To rotate a query or key within one attention head:

  • Group its dimensions into two-dimensional pairs. A four-dimensional vector [x1, x2, x3, x4] becomes (x1, x2) and (x3, x4). section 3.11.2
  • Rotate each pair by an angle determined by the token’s position; different pairs use different angles. For a pair (x, y), rotation by θ produces (x cos θ − y sin θ, x sin θ + y cos θ). section 3.11.2
  • Reassemble the rotated pairs into the position-encoded vector. Apply the same process to keys before calculating attention scores. section 3.11.2 section 3.12
Figure 3.25 Each pair of dimensions is treated as a 2D vector and rotated by a position-dependent angle θ.

Visualize each dimension pair as a two-dimensional vector rotated by a position-dependent angle. section 3.11.2

Position matters in the query–key interaction: their dot product determines an attention score. RoPE puts the positional transformation at that interaction rather than adding it to token embeddings before the Transformer block. section 3.10.3 section 3.11.1

Standard RoPE creates a problem for Multi-Head Latent Attention (MLA). MLA’s absorption trick combines position-agnostic weight matrices in advance and caches a compressed key–value representation. A position-dependent rotation between the matrices prevents that fixed combination; recomputing full rotated historical keys at every generation step would undermine caching efficiency. section 3.13

Decoupled RoPE separates the attention calculation into a content path and a position path. The content path uses MLA without positional information, retaining absorption; the position path uses separate projections with RoPE. Their scores are added to produce the final attention scores. section 3.14

Updating the content cache

For a new token, an embedding of shape (1, 8) is multiplied by a down-projector of shape (8, 4), producing a latent vector of shape (1, 4). Appending it to a (4, 4) latent KV cache gives a (5, 4) cache. An absorbed content query then scores against the updated cache while the position path runs in parallel. section 3.14.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains pairwise rotations and the matrix-absorption conflict, then outlines separate content and position paths.
Examples
Uses a four-dimensional vector to illustrate pairing and traces a new token through a latent-cache update.
What you can do after
Enables readers to evaluate how position-dependent rotations affect attention and why MLA uses a decoupled position path.
How it connects
Connects positional encoding with sinusoidal embeddings, attention scores, KV caching, and Multi-Head Latent Attention.

Also explained in

Compare how 5 books explain Rotary Position Embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.