liveBook · Manning

Binary positional encoding

As explained in Build a DeepSeek Model (From Scratch)

Binary positional encoding represents sequence positions using binary-valued positional codes rather than unbounded integer values.

Binary positional encoding represents sequence positions with bounded binary-valued codes, avoiding the large, unbounded magnitudes of integer positional encodings that can overwhelm semantic token embeddings. section 3.9

Each bit contributes a different positional resolution. Low-order bits flip rapidly and distinguish nearby positions; high-order bits flip slowly and convey coarse separation across distant positions. section 3.9.2

Figure 3.14 The oscillating frequency of bits in binary positional encodings. Lower indices (like the LSB) oscillate rapidly across positions, while higher indices (like the MSB) oscillate slowly.

Use the bit-oscillation figure to inspect how binary positions decompose into fast fine-grained and slow coarse-grained signals. section 3.9.2

Positions 64 through 75

Across positions 64–75, the least significant bit alternates at every step, the next bit changes every two positions, and higher bits change progressively more slowly. The representation therefore supplies both local and global positional cues. section 3.9.2

The bounded representation exposes a useful architectural trade-off:

  • Preserves unique positions without unbounded numeric magnitude. section 3.9
  • Produces multi-frequency signals that separate adjacent and distant positions. section 3.9.2
  • Introduces abrupt 0-to-1 square-wave transitions, creating discontinuities that hinder gradient-based optimization. section 3.9.3
  • Motivates sinusoidal positional encoding, which retains multiple frequencies while replacing discrete jumps with smooth sine and cosine waves. section 3.10 section 3.10.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains bounded values, bit frequencies, concrete positions, and optimization trade-offs before deferring to sinusoidal encodings.
Examples
Uses positions 64–75 and bit-oscillation plots to show local and global positional resolution.
What you can do after
Evaluate/understand why binary codes solve magnitude problems yet motivate smooth sinusoidal alternatives.
How it connects
Connects integer positional encoding, token embeddings, Transformers, gradient-based optimization, and sinusoidal positional encoding.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.