liveBook · Manning

RotaryPositionalEncoding

As explained in Build a DeepSeek Model (From Scratch)

RotaryPositionalEncoding is a module that generates and applies rotary positional encodings to query and key representations.

RotaryPositionalEncoding generates rotation frequencies and applies them to query and key representations, injecting positional information by rotating pairs of dimensions. It precomputes frequency values and supports position offsets for cached inference, where each step processes a new token. section 6.2.1

The implementation follows a compact precompute-and-apply pipeline:

  • Slices rotations using position_offset so cached tokens and newly processed tokens receive aligned positions. section 6.2.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains frequency precomputation, complex rotation, tensor application, and cached-inference offsets through an implementation listing.
Examples
Uses a PyTorch RotaryPositionalEncoding module and integrates it into DeepSeekAttention's separate position pathway.
What you can do after
Build it from scratch as a PyTorch helper for training and position-aligned cached inference.
How it connects
Connects positional encoding to query vectors, key vectors, attention, complex number multiplication, and key-value caching.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.