Attention lets a neural network decide which parts of its input matter most to a particular part. In a Transformer, self-attention gives each token a way to weigh other tokens when building a context-aware representation. Think of reading with a highlighter: when you reach “tired” in “Alice was beginning to get very tired,” you look back to “Alice” for who is tired and “very” for how tired. section 11.17
Why use it? An RNN compresses preceding text into a fixed-size memory vector, which can lose earlier context. A CNN detects local word patterns but is less suited to long-range meaning. A Transformer processes a sequence simultaneously; self-attention lets tokens weigh one another. Because that operation alone does not identify word order, positional encodings add position information to token embeddings. section 11.11 section 11.15
For one attention head, the computation follows four steps:
- Project each input embedding through three learned weight matrices to produce a Query (Q), Key (K), and Value (V). A Query represents what a token seeks, a Key helps identify a match, and a Value supplies what a matching token contributes. section 11.18
- Compare each Query with the Keys using dot products, then scale the scores by the square root of the dimension. section 11.21
- Apply softmax to turn the scores into attention weights. section 11.21
- Multiply each Value by its weight and add the results. The resulting context vector blends information from the tokens receiving attention. section 11.21
A small context-vector calculation
In a 32-token, four-value-wide example, one token assigns weights of 0.0632 to token 0 and 0.4064 to token 31. Multiplying every token’s Value vector by its weight and summing produces the first token’s context vector: [0.2283, 2.1990, 0.4526, 1.9747]. The larger weight makes token 31’s Value contribute more strongly to that sum. section 11.21
Figure 11.34 An analogy of the purposes of Query, Key, and Value
The ingredient analogy separates what a Query seeks, what a Key advertises, and what a Value contributes. section 11.18
A head offers one perspective on token relationships; multiple heads can notice different patterns. Stacked Transformer blocks repeat attention with their own learned weights, refining representations before next-token prediction. Attention also connects language to images: in a U-Net, cross-attention uses image features as Queries and text embeddings as Keys and Values, helping image regions respond to relevant prompt words. section 11.17 section 11.28 section 12.29
