Why would a Transformer need to be told where words occur? Unlike a recurrent model, it does not step through a sentence in order; unlike a convolutional model, it does not sweep across neighboring words. Without an added position signal, self-attention cannot distinguish “cat fat” from “fat cat.” A positional encoding supplies that signal alongside each word embedding before self-attention begins. section 5.1
The original Transformer, introduced in 2017, used sinusoidal functions at different frequencies. These fixed encodings let the model infer absolute positions and relative distances while attention processes the sequence in parallel. Position did not require bringing back recurrence. section 5.0 section 5.1
Later Transformers changed how the position signal is represented:
- BERT used learnable positional embeddings instead of fixed sinusoids, allowing training to shape the representations. section 5.1
- GPT-2 and GPT-3 used learned absolute positional embeddings; T5 used learned relative embeddings. section 5.1
- LLaMA used rotary positional embeddings to inject relative position into attention itself. section 5.1
Figure 5.1 A simplified schematic of the Transformer architecture. The model consists of an encoder (left) and a decoder (right), each comprising repeated stacks of sublayers, including multihead attention mechanisms, feed-forward neural networks, and postprocessing steps such as residual connections and layer normalization. Positional encodings (depicted as circled crosses) are added to the input embeddings to explicitly represent token order. The diagram emphasizes the cross-attention operation (dashed box labeled “Cross-attention”), in which the decoder’s queries (Q) interact with the encoder’s keys (K) and values (V). During cross-attention, each decoding step computes attention scores to determine the relevance of encoded input representations, producing a context vector that guides token generation. This selective alignment enables the decoder to integrate contextual information from the encoder, improving the accuracy and coherence of the output sequence.
The circled crosses show where positional encodings join input embeddings before the Transformer’s attention layers. section 5.1
The solution did not end the argument about order. Critics saw positional encodings as a workaround for missing recurrence, and word-reordering tests found that self-attention could struggle with positional cues even when embeddings were present. Yet Transformers trained for word-order-sensitive machine translation outperformed recurrent models at learning positional information. The engineering lesson is narrower than either slogan: adding a position signal matters, but performance also depends on the task and training. section 5.1.2
