A pronoun can depend on a noun several words away. Self-attention constructs a representation of each token by weighting information from other tokens in the same sequence, so its representation changes with context. Unlike a recurrent hidden state, these relationships need not pass through every intervening state update. section 12.1.2 section 12.2
Each token’s embedding is transformed by learned projections into a query, a key, and a value. The query expresses what the token seeks; keys signal relevance; values carry information to be combined. For a query q and key kᵢ, a scaled dot product gives a compatibility score sᵢ = (q · kᵢ) / √d, where d is the vector dimension. Softmax turns the scores into weights αᵢ = exp(sᵢ) / Σⱼ exp(sⱼ). The contextual representation is z = Σᵢ αᵢvᵢ: a weighted sum of the values. section 12.2.1
Resolving a pronoun
In a simplified example, the query for “it” is [0.2, 0.8], while the keys and values for “trophy” and “box” are [1, 0] and [0, 1]. Their dot products with the query are 0.2 and 0.8. After scaling and softmax, the weights are approximately 0.39 and 0.61, producing [0.39, 0.61]. The representation draws more strongly on “box.” These two-dimensional vectors and identity projections make the calculation transparent; they do not reproduce a full trained model. section 12.2.1 section 12.2
Self-attention is the central operation of the transformer, but it works alongside other components. Positional information distinguishes order, multiple heads can capture different relationships, and feed-forward networks further transform representations. In a decoder, causal masking prevents self-attention from accessing future tokens. section 12.1.2 section 12.1.2
Figure 12.1 The transformer architecture introduced in Attention Is All You Need. Stacked encoder and decoder layers combine multi-head attention with feed-forward transformations, residual connections, and normalization. Positional encodings preserve sequence order without recurrence. The decoder uses masked self-attention to prevent access to future tokens and cross-attention to incorporate encoder outputs. Within each multi-head attention block, individual heads use dot products to calculate compatibility scores and softmax to convert those scores into attention weights, as demonstrated in the worked examples that follow. Reproduced from Vaswani et al (2017), Attention Is All You Need, Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017).

The architecture diagram locates self-attention within encoder and decoder layers and shows how masking and cross-attention differ. section 12.1.2
Attention operations can be computed largely in parallel during training, unlike recurrent state propagation. This supports scale, but context length, model capacity, training data, and computation still constrain what a model can use. Autoregressive language models generate tokens one at a time, and factual recall depends on learned components throughout the network—not on one attention calculation alone. section 10.5.2 section 12.2.2 section 12.6




