Sinusoidal positional encoding gives each sequence position a bounded, deterministic vector fingerprint. Add this vector to each residue embedding before the first encoder layer so self-attention can use position as well as content. section 12.3.2
The representation works through several complementary signals:
- Use sine and cosine functions across embedding dimensions, with different frequencies for different dimensions. section 12.3.2
- Let lower dimensions oscillate rapidly and higher dimensions oscillate slowly, like the bits of a binary counter or the hands of a clock. section 12.3.2
- Combine these waves to produce a distinct vector for each position and smooth relationships between relative offsets. section 12.3.2
Figure 12.10 (Left) A discrete binary counter (black/white), showing how lower bits (towards the left on the x-axis) oscillate rapidly while higher bits (towards the right on the x-axis) oscillate slowly. (Right) The Transformer's sinusoidal encoding (red/blue), showing the exact same frequency gradient but in a continuous, differentiable form, using sine and cosine waves of different frequencies to assign a unique, differentiable vector fingerprint to every position in the sequence.

Compare binary counting with continuous sinusoidal waves to see how changing frequencies encode sequence position. section 12.3.2
