Binary positional encoding represents sequence positions with bounded binary-valued codes, avoiding the large, unbounded magnitudes of integer positional encodings that can overwhelm semantic token embeddings. section 3.9
Each bit contributes a different positional resolution. Low-order bits flip rapidly and distinguish nearby positions; high-order bits flip slowly and convey coarse separation across distant positions. section 3.9.2
Figure 3.14 The oscillating frequency of bits in binary positional encodings. Lower indices (like the LSB) oscillate rapidly across positions, while higher indices (like the MSB) oscillate slowly.

Use the bit-oscillation figure to inspect how binary positions decompose into fast fine-grained and slow coarse-grained signals. section 3.9.2
Positions 64 through 75
Across positions 64–75, the least significant bit alternates at every step, the next bit changes every two positions, and higher bits change progressively more slowly. The representation therefore supplies both local and global positional cues. section 3.9.2
The bounded representation exposes a useful architectural trade-off:
- Preserves unique positions without unbounded numeric magnitude. section 3.9
- Produces multi-frequency signals that separate adjacent and distant positions. section 3.9.2
- Introduces abrupt 0-to-1 square-wave transitions, creating discontinuities that hinder gradient-based optimization. section 3.9.3
- Motivates sinusoidal positional encoding, which retains multiple frequencies while replacing discrete jumps with smooth sine and cosine waves. section 3.10 section 3.10.1
