liveBook · Manning

Relative position

As explained in Build a DeepSeek Model (From Scratch)

A relative positional relationship describes the positional difference between tokens rather than representing each absolute position independently.

Relative position describes how tokens are separated, rather than assigning each position an independent representation. Sinusoidal encodings turn this difference into a fixed rotation: moving from position p to p+k rotates the positional vector by k·ωᵢ, giving the model a smooth, reusable positional relationship. section 3.10.2

θ = p / 10000^(2i / d_model)

Use the angle formula θ = p / 10000^(2i / d_model) to connect position p, dimension pair i, and the rotation applied to the corresponding sinusoidal coordinates. section 3.10.2

Rotary positional encoding applies the relative-position idea directly to attention:

  • Pairs dimensions of each Query and Key vector, treats every pair as a two-dimensional vector, and rotates it by a position-dependent angle. section 3.11.2
  • Preserves vector magnitude because rotation changes direction without changing length, injecting positional information without distorting learned semantic representations. section 3.11.2
  • Makes the attention score between a Query at m and a Key at n depend on their relative distance m−n through the difference between rotation angles. section 3.11.2
  • Combines direct Query/Key modification with multi-frequency signals, capturing both fine-grained and coarse-grained positional relationships. section 3.11.2

A two-step relationship

To relate tokens at positions p and p+2, apply the same two-step rotation to the positional vector at p. For attention, the analogous rotation difference between positions m and n represents m−n. section 3.10.2 section 3.11.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains rotation geometry, magnitude preservation, angle differences, and multi-frequency signals, stopping short of implementation code.
Examples
Uses unit-circle vectors, position p plus k, a two-step rotation, and Query-Key positions m and n.
What it assumes
Assumes familiarity with token positions, vectors, attention, and sine-cosine identities while grounding the geometric rotation intuition.
What you can do after
Evaluate how rotation encodes relative distance while preserving vector magnitude in attention representations.
How it connects
Connects sinusoidal positional encoding, rotary positional encoding, attention, Query and Key vectors, and token embeddings.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.