Rotary Position Embedding (RoPE) encodes token positions by applying position-dependent geometric rotations to representations. It supplies positional information within Transformer blocks, where token embeddings, self-attention, feed-forward layers, residual connections, and normalization work together. This preserves the Transformer’s ability to process tokens in parallel while distinguishing their locations in a sequence. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Trace the input path from subword tokens and integer IDs through token embeddings and positional encoding before the first Transformer block. section 3.1.1




