Positional information supplies token order to Transformer models, which do not naturally know whether a token comes first, second, or later. It is combined with token embeddings before the first Transformer block. section 3.1.1
Common approaches differ in adaptability and length behavior:
- Fixed positional patterns use sine and cosine functions. section 3.1.1
- Learned positional representations adapt during training but are limited by the sequence lengths seen during training. section 3.1.1
- Rotary position embeddings (RoPE) encode positions through geometric rotations to represent relative order efficiently for longer texts. section 3.1.1
Figure 3.3 The input pipeline before the Transformer block including the positional embedding. Raw text is tokenized into subword units and mapped to integer IDs, which are then looked up in a learned embedding matrix to produce dense token vectors. Positional embeddings encoding each token's position in the sequence are added to the token embeddings, producing the final input matrix passed to the first Transformer block.

Follow the input pipeline from raw text and subword tokenization through token embeddings, positional embeddings, and the matrix entering the first Transformer block. section 3.1.1









