Positional embedding, also called positional encoding, supplies order information to token representations. A Transformer processes tokens in parallel, so token embeddings alone do not intrinsically indicate which word came first, next, or last. Adding a distinct positional code gives the model a map of the sequence while retaining parallel processing. section 2.2
The encoding follows the input embedding step: tokens first become dense vectors, then receive positional information. This lets the model distinguish the sequential arrangement of words, an essential part of language understanding that recurrence would otherwise provide through sequential processing. section 2.2
Within Transformer models, positional embeddings work alongside other components:
- Self-attention relates words across a sequence, while positional encoding keeps track of their order. section 2.5
- Multi-head attention provides broader views of token relationships; positional information preserves where those relationships occur in the sequence. section 2.5
- Decoder models use token embeddings, positional encodings, decoder layers, masked self-attention, and feed-forward networks for sequential generation. section 5.1
Autoregressive next-token generation
A decoder-only language model processes preceding tokens to predict the next token. Positional encodings maintain the order of those earlier tokens as embeddings pass through decoder blocks, supporting causal generation with masked self-attention. section 5.1
Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.
The diagram places positional encoding between token embedding and decoder blocks in an autoregressive next-word prediction flow. section 5.1



















