Rotary Position Embedding (RoPE) injects position into transformer features by rotating pairs of values. Treat each pair as a complex number: at position m, pair j rotates by angle mωj, with frequencies decreasing geometrically across feature pairs. Applying position-dependent rotations to query and key vectors makes their dot-product relationship depend predictably on the distance between positions. section 3.5.1
RoPE rotation formula
The rotation matrix shows how each feature pair becomes two transformed coordinates. section 3.5.1
On a GPU, the key task is translating each logical pair into memory offsets. In the shared-frequency layout, input x advances by feature_dim, while the cos and sin tables advance by feature_dim // 2. Work through concrete indices before coding; check that each output is written exactly once and that separate program instances do not collide. Triton provides block-level offsets, but the mapping from algorithm coordinates to data layout remains your responsibility. section 3.5 section 3.5.4
Some layouts use a distinct rotation angle for every feature rather than sharing one across each pair. Their cosine and sine tables have shape [seq_len, feature_dim], so a kernel must load separate values for the two elements. Consider how this layout changes memory intensity when adapting the kernel. section 3.6.3




