Self-attention lets a Transformer build a contextual representation of each token by selectively drawing information from other tokens in the same sequence. Which tokens matter depends on the token being interpreted, much as different parts of a photograph matter when identifying an object versus its surroundings. Unlike a single stream of memory passed from left to right, attention can connect tokens across a sequence, subject to any restrictions imposed on it. section 4.1.3 section 4.1.6
Each token representation is transformed through three learned projections into a query, a key, and a value. The query expresses what the token seeks in context; the key describes characteristics used to judge relevance; and the value carries information the token can contribute. Comparing a token’s query with other tokens’ keys produces relevance scores. Their values are then combined according to the resulting attention pattern, giving the token a representation that incorporates context. section 4.1.3
Interpreting “rises”
In “The sun rises in the east,” the query for “rises” can be compared with the keys of the other tokens. A stronger match with “sun” than with “the” gives “sun” a higher relevance score, so its value contributes more to the contextual representation of “rises.” section 4.1.3
Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

The diagram traces learned projections, attention scores, and the combination of values into contextual representations. section 4.1.3
Attention does not itself encode token order, so positional information must be supplied explicitly. Multi-head attention provides concurrent views of the input, allowing different relational patterns to be captured. Self-attention is one part of a Transformer block, alongside feedforward layers and supporting connections and normalization. section 4.8 section 4.1.6
The permitted connections depend on the task. An encoder can let every token attend to every other token in the input. For step-by-step generation, a decoder uses a causal mask: each position can attend only to itself and earlier positions. This prevents a prediction from accessing future tokens. The attention mechanism remains the same; the mask changes which interactions are allowed. section 4.1.6
Figure 4.10 Unmasked and masked self-attention. In these matrices, darker cells indicate stronger attention between tokens, while hatched cells mark positions that are masked. In unmasked attention (left), each token can attend to any other token in the sequence. In masked attention (right), future positions are hidden, allowing each token to see only itself and earlier tokens. This causal structure enables step-by-step text generation, where each new token is produced using only what came before.

The matrices contrast unrestricted attention with causal masking of future positions. section 4.1.6




