Relative position describes how tokens are separated, rather than assigning each position an independent representation. Sinusoidal encodings turn this difference into a fixed rotation: moving from position p to p+k rotates the positional vector by k·ωᵢ, giving the model a smooth, reusable positional relationship. section 3.10.2
Use the angle formula θ = p / 10000^(2i / d_model) to connect position p, dimension pair i, and the rotation applied to the corresponding sinusoidal coordinates. section 3.10.2
Rotary positional encoding applies the relative-position idea directly to attention:
- Pairs dimensions of each Query and Key vector, treats every pair as a two-dimensional vector, and rotates it by a position-dependent angle. section 3.11.2
- Preserves vector magnitude because rotation changes direction without changing length, injecting positional information without distorting learned semantic representations. section 3.11.2
- Makes the attention score between a Query at m and a Key at n depend on their relative distance m−n through the difference between rotation angles. section 3.11.2
- Combines direct Query/Key modification with multi-frequency signals, capturing both fine-grained and coarse-grained positional relationships. section 3.11.2
A two-step relationship
To relate tokens at positions p and p+2, apply the same two-step rotation to the positional vector at p. For attention, the analogous rotation difference between positions m and n represents m−n. section 3.10.2 section 3.11.2
