liveBook · Manning

Positional embedding

As explained in Build a DeepSeek Model (From Scratch)

Positional encoding is a technique that represents a token's sequence position and incorporates that information into its semantic representation.

Attention needs positional information to distinguish where tokens occur in a sequence. A positional embedding represents a token’s position and can be added to its semantic embedding before the first Transformer block. The challenge is to make position useful without overwhelming meaning. section 3.7 section 3.10.3

Two occurrences of the same token

In “The dog chased another dog,” the two occurrences of “dog” have the same token embedding but appear at positions 2 and 5. A simple eight-dimensional scheme adds [2, 2, 2, 2, 2, 2, 2, 2] to the first and [5, 5, 5, 5, 5, 5, 5, 5] to the second. This distinguishes them, but larger position numbers can dominate the relatively small semantic values. section 3.8.1 section 3.8.2

Binary encoding bounds each coordinate to 0 or 1; for example, position 200 becomes [1, 1, 0, 0, 1, 0, 0, 0] in eight bits. Its coordinates change at different frequencies, but their abrupt jumps motivate a smoother positional signal. section 3.9.1 section 3.9.3

Sinusoidal encoding replaces those jumps with smooth waves between −1 and 1. For position p, dimension pair i, and embedding size d, calculate PE(p, 2i) = sin(p / 10000^(2i / d)) and PE(p, 2i + 1) = cos(p / 10000^(2i / d)). Build a vector across the dimension pairs, then add it to the token embedding. Lower-index pairs oscillate faster; higher-index pairs change more slowly. section 3.10.1

Figure 3.18 Visualization of sinusoidal positional encodings for a GPT-2-sized model. The plots show the encoding values for different indices (i) across 1024 positions.

Compare fast and slow positional waves across 1024 positions. section 3.10.1

Each sine–cosine pair can also be viewed as coordinates on a circle. Moving from position p to p + k rotates that pair by an angle determined by k and the pair’s frequency. Thus, absolute position encodings have a consistent linear relationship for a given relative offset. section 3.10.2

Figure 3.20 The positional encoding vector v2 for a future position p + k is a simple rotation of the original vector v1 for position p.

Visualize the rotation connecting two positions separated by k steps. section 3.10.2

Addition still mixes positional and semantic signals, although Transformer layers can learn to reinterpret them. Rotary positional encoding takes a different path: it rotates query and key vectors inside attention according to position. Rotation preserves their magnitudes, though it changes their directions. section 3.10.3 section 3.11.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops integer, binary, and sinusoidal schemes through formulas, frequency patterns, and relative-position rotations, then contrasts additive encoding with RoPE.
Examples
Uses repeated instances of “dog,” an eight-bit encoding of position 200, and sinusoidal plots over 1024 positions.
What you can do after
Evaluate and understand positional encoding designs by comparing their magnitude, continuity, and integration trade-offs.
How it connects
Connects token embeddings, attention, relative position, and rotary positional encoding.

Also explained in

Compare how 20 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.