liveBook · Manning

Sinusoidal positional encoding

As explained in Build AI Drug Discovery Pipelines

A sinusoidal position encoding represents sequence positions with fixed sine and cosine functions added to model inputs.

Sinusoidal positional encoding gives each sequence position a bounded, deterministic vector fingerprint. Add this vector to each residue embedding before the first encoder layer so self-attention can use position as well as content. section 12.3.2

The representation works through several complementary signals:

  • Use sine and cosine functions across embedding dimensions, with different frequencies for different dimensions. section 12.3.2
  • Let lower dimensions oscillate rapidly and higher dimensions oscillate slowly, like the bits of a binary counter or the hands of a clock. section 12.3.2
  • Combine these waves to produce a distinct vector for each position and smooth relationships between relative offsets. section 12.3.2
Figure 12.10 (Left) A discrete binary counter (black/white), showing how lower bits (towards the left on the x-axis) oscillate rapidly while higher bits (towards the right on the x-axis) oscillate slowly. (Right) The Transformer's sinusoidal encoding (red/blue), showing the exact same frequency gradient but in a continuous, differentiable form, using sine and cosine waves of different frequencies to assign a unique, differentiable vector fingerprint to every position in the sequence.

Compare binary counting with continuous sinusoidal waves to see how changing frequencies encode sequence position. section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains bounded fingerprints, frequency intuition, and relative offsets, but omits detailed derivation and implementation code.
Examples
Uses protein sequences, residue positions, binary counters, clock hands, sine waves, and a visual frequency comparison.
What it assumes
Assumes sequence embeddings and encoder layers, while defining why raw integer positions are unsuitable.
What you can do after
Apply it by adding fixed sine-and-cosine position vectors to sequence embeddings before encoder processing.
How it connects
Connects positional encoding with residue embeddings, encoder-only transformers, and multi-head self-attention.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.