liveBook · Manning

Rotation

As explained in Build a DeepSeek Model (From Scratch)

A rotation is a geometric transformation that changes the orientation of a vector by a specified angle while preserving its length.

Rotation changes a vector’s orientation while preserving its length. In sinusoidal positional encoding, each even–odd dimension pair forms a 2D vector on a unit circle. Moving from position p to p + k rotates that vector by an angle determined by k and the pair’s frequency, turning relative position into a simple geometric relationship. section 3.10.2

θ = p / 10000^(2i / d_model)

The rotation angle begins with the position-dependent frequency θ = p / 10000^(2i / d_model); the same frequency determines how far a dimension pair rotates as position changes. section 3.10.2

Rotation inside attention

Rotary Positional Encoding applies rotation directly to Query and Key vectors inside attention instead of adding positional vectors to token embeddings. This injects position where attention uses it, while preserving the learned magnitude of each vector; only its orientation changes. section 3.11.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains 2D geometry, trigonometric position shifts, and attention placement, but stops short of implementation details.
Examples
Uses unit-circle dimension pairs and the shift from position p to p + k to illustrate rotation.
What it assumes
Assumes familiarity with vectors, token embeddings, Query and Key representations, and attention.
What you can do after
Evaluate/understand how rotation encodes position while preserving vector magnitude inside attention.
How it connects
Connects sinusoidal positional encoding, relative positional relationships, token embeddings, attention, and Rotary Positional Encoding.

Also explained in

Compare how 5 books explain rotation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.