In this RoPE layout, each feature pair, xᵢ and xᵢ₊D⁄₂, uses the same rotation angle. Cosine and sine tables therefore store one value per pair, with shape [sequence length, D/2]. section 3.5.2 section 3.6.3
A Triton kernel must map each sequence row into two different layouts:
- Load paired input features across the midpoint of a row of width D. section 3.5.2
- Load the shared cosine and sine values from the corresponding row of width D/2; keeping the row offsets separate avoids mixing the layouts. section 3.5.3
- Reuse each angle pair for two input elements, reducing auxiliary-table loads compared with a full elementwise design. section 3.5.3 section 3.6.3
Figure 3.9 A tensor with shape [4,8] stored in row-major order. The 2D structure (top) maps to a flat memory layout (bottom) where all elements of each row are stored consecutively. Within each row, element i pairs with element i+4 for the rotation operation. The pairing arrows show how positions 0 and 4, 1 and 5, 2 and 6, 3 and 7 form the four pairs for Position 0.
The diagram traces how a width-eight row pairs positions 0–3 with positions 4–7. section 3.5.2
