liveBook · Manning

Rotary Position Embedding

As explained in GPU Programming with Triton

Rotary Positional Embeddings encode position by applying position-dependent rotations to pairs of neural-network features.

Rotary Position Embedding (RoPE) injects position into transformer features by rotating pairs of values. Treat each pair as a complex number: at position m, pair j rotates by angle mωj, with frequencies decreasing geometrically across feature pairs. Applying position-dependent rotations to query and key vectors makes their dot-product relationship depend predictably on the distance between positions. section 3.5.1

RoPE rotation formula
\[\begin{bmatrix} x'_1 \\ x'_2 \end{bmatrix} = \begin{bmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\]

The rotation matrix shows how each feature pair becomes two transformed coordinates. section 3.5.1

On a GPU, the key task is translating each logical pair into memory offsets. In the shared-frequency layout, input x advances by feature_dim, while the cos and sin tables advance by feature_dim // 2. Work through concrete indices before coding; check that each output is written exactly once and that separate program instances do not collide. Triton provides block-level offsets, but the mapping from algorithm coordinates to data layout remains your responsibility. section 3.5 section 3.5.4

Some layouts use a distinct rotation angle for every feature rather than sharing one across each pair. Their cosine and sine tables have shape [seq_len, feature_dim], so a kernel must load separate values for the two elements. Consider how this layout changes memory intensity when adapting the kernel. section 3.6.3

How GPU Programming with Triton covers it

How deep it goes
Derives pairwise rotations and frequency scaling, then works through shared-layout indexing and a full-layout kernel variant.
Examples
Contrasts reordered sentences to motivate position and shared-frequency versus full elementwise layouts.
What you can do after
Build it from scratch: map paired-feature loads and stores to sequence positions, then adapt indexing for distinct per-element rotations.
How it connects
Connects positional encoding to Transformers, query-key dot products, and Triton coordinate mapping.

Also explained in

Compare how 5 books explain Rotary Position Embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.