Attention lets a neural network weigh relationships among inputs so that relevant information contributes more strongly to a result. Think of highlighting important words while reading: the useful words can change depending on what you are trying to understand. section 11.1
In a Transformer, self-attention lets each token—a word or word fragment—draw information from other tokens in the sequence. For next-token prediction, the model can focus on relevant earlier tokens. Attention heads provide different perspectives on relationships, while stacked Transformer blocks can refine representations further. section 11.11 section 11.1 section 11.28
Following a relationship
In “Alice was beginning to get very tired,” understanding “tired” benefits from connecting it to “Alice,” who is tired, and “very,” which describes how tired. Self-attention gives tokens a way to weight such relationships when forming their representations. section 11.17
A single scaled dot-product attention head works in small steps:
- Project each token’s embedding—a vector representing the token—into a query, key, and value using learned weight matrices. The query looks for relevant information, the key helps identify it, and the value carries the contribution. section 11.18
- Compare each query with keys using dot products. Scale the scores by the square root of the vector dimension, then use softmax to turn them into attention weights. section 11.21
- Multiply each value by its attention weight and add the results. The resulting context vector blends information for the querying token. section 11.21
A weighted contribution
In a four-value example, one token has an attention weight of 0.0632 and a value whose second component is 1.9940. Its contribution to that component is about 0.1261. Adding the corresponding contributions from all tokens produces the context vector. section 11.21
Self-attention alone does not tell the model token order. Positional encoding adds a position signal to each embedding, helping distinguish sentences such as “The dog bit the man” and “The man bit the dog.” section 11.15
Attention also connects language to images. In a U-Net, cross-attention can use image features as queries and a text embedding as keys and values. As the image is refined, its regions can draw on relevant parts of a prompt—for example, the word “snail” for a rounded patch. section 12.29
Figure 12.56 How cross-attention fits into the U-Net architecture
See where cross-attention fits between the U-Net’s downsampling and upsampling layers. section 12.29




























