A pronoun may refer to a noun many words earlier. A recurrent network carries that context through successive hidden states; a transformer can instead relate the tokens directly. Introduced in 2017, the transformer replaced recurrent sequence processing with self-attention, making relational context its organizing principle and allowing much of training to run in parallel. section 12.0 section 12.1.2
For a token, self-attention compares its query q with other tokens’ keys kᵢ. Softmax turns the resulting compatibility scores into normalized weights, αᵢ = exp(q·kᵢ) / Σⱼ exp(q·kⱼ). The contextual representation is then Σᵢ αᵢvᵢ, a weighted combination of value vectors vᵢ. The weights express how strongly the compared tokens contribute; they need not be equal, even for tokens far apart. section 10.5.2 section 12.1.2
Attention is one part of a larger architecture. Positional information distinguishes sequence order; multiple attention heads capture different relational patterns; and feed-forward networks further transform representations. Residual connections and normalization support information flow and stable training across stacked layers. section 12.1.2
Figure 12.1 The transformer architecture introduced in Attention Is All You Need. Stacked encoder and decoder layers combine multi-head attention with feed-forward transformations, residual connections, and normalization. Positional encodings preserve sequence order without recurrence. The decoder uses masked self-attention to prevent access to future tokens and cross-attention to incorporate encoder outputs. Within each multi-head attention block, individual heads use dot products to calculate compatibility scores and softmax to convert those scores into attention weights, as demonstrated in the worked examples that follow. Reproduced from Vaswani et al (2017), Attention Is All You Need, Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017).

The original encoder–decoder design shows how attention, feed-forward transformations, positional encodings, residual connections, and normalization fit together. Its decoder uses masked self-attention and cross-attention to encoder outputs. section 12.1.2
From context to prediction
In pronoun resolution, attention can weight candidate words to construct a context-sensitive representation before prediction. Related calculations support semantic association; causal masking restricts access to future tokens during autoregressive generation. A simplified attention calculation does not reproduce the combined behavior of a full model with multiple heads and layers. section 12.2 section 12.1.2
The architecture changes information flow, not the basic learning framework: predictions are evaluated and parameters updated through backpropagation and gradient-based optimization. Parallel training does not mean parallel generation; transformer language models generate one token at a time. Performance also depends on data, capacity, objectives, and computation. Finite context, computational cost, interpretability, and factual reliability remain constraints, and fluent output can be wrong. section 12.1 section 12.1.3 section 12.5 section 12.6







































