Attention needs positional information to distinguish where tokens occur in a sequence. A positional embedding represents a token’s position and can be added to its semantic embedding before the first Transformer block. The challenge is to make position useful without overwhelming meaning. section 3.7 section 3.10.3
Two occurrences of the same token
In “The dog chased another dog,” the two occurrences of “dog” have the same token embedding but appear at positions 2 and 5. A simple eight-dimensional scheme adds [2, 2, 2, 2, 2, 2, 2, 2] to the first and [5, 5, 5, 5, 5, 5, 5, 5] to the second. This distinguishes them, but larger position numbers can dominate the relatively small semantic values. section 3.8.1 section 3.8.2
Binary encoding bounds each coordinate to 0 or 1; for example, position 200 becomes [1, 1, 0, 0, 1, 0, 0, 0] in eight bits. Its coordinates change at different frequencies, but their abrupt jumps motivate a smoother positional signal. section 3.9.1 section 3.9.3
Sinusoidal encoding replaces those jumps with smooth waves between −1 and 1. For position p, dimension pair i, and embedding size d, calculate PE(p, 2i) = sin(p / 10000^(2i / d)) and PE(p, 2i + 1) = cos(p / 10000^(2i / d)). Build a vector across the dimension pairs, then add it to the token embedding. Lower-index pairs oscillate faster; higher-index pairs change more slowly. section 3.10.1
Figure 3.18 Visualization of sinusoidal positional encodings for a GPT-2-sized model. The plots show the encoding values for different indices (i) across 1024 positions.

Compare fast and slow positional waves across 1024 positions. section 3.10.1
Each sine–cosine pair can also be viewed as coordinates on a circle. Moving from position p to p + k rotates that pair by an angle determined by k and the pair’s frequency. Thus, absolute position encodings have a consistent linear relationship for a given relative offset. section 3.10.2
Figure 3.20 The positional encoding vector v2 for a future position p + k is a simple rotation of the original vector v1 for position p.

Visualize the rotation connecting two positions separated by k steps. section 3.10.2
Addition still mixes positional and semantic signals, although Transformer layers can learn to reinterpret them. Rotary positional encoding takes a different path: it rotates query and key vectors inside attention according to position. Rotation preserves their magnitudes, though it changes their directions. section 3.10.3 section 3.11.1



















