Attention assigns learned weights to relationships between elements in a sequence. A high weight means one element is more relevant to another; in self-attention, each element can also attend to itself. This helps a Transformer connect words far apart in a sentence. Unlike an RNN, which processes a sequence one element at a time, a Transformer can process inputs in parallel during training. section 1.3.1 section 1.3
From a sentence to attention
For “How are you?”, tokenize the sentence as [how, are, you, ?] and represent each token with a 256-dimensional embedding. Add positional encoding so the input also carries token order. The resulting 4 × 256 tensor supplies the representations whose relationships self-attention calculates. section 9.1.1
Scaled dot-product attention turns those representations into weighted information:
- Pass input embeddings X through learned projections to produce queries Q, keys K, and values V. Think of a library search: the search phrase is a query, book descriptions are keys, and recommended books are values. section 9.1.1
- Compare each query with the keys using a dot product, then divide the scores by the square root of the key dimension. Apply softmax to obtain weights that sum to 100% for each querying token. section 9.1.1 section 9.5
- Multiply the weights by V to form attention vectors. Multihead attention splits Q, K, and V into parallel heads, letting the model attend to different aspects of the input before combining their outputs. section 9.1.1 section 9.5
Figure 9.1 A diagram of the self-attention mechanism. To calculate attention, the input embedding X is first passed through three neural layers with weights, WQ, WK, and WV, respectively. The outputs are query Q, key K, and value V. The scaled attention score is the product of Q and K divided by the square root of the dimension of K, dk. We apply the softmax function on the scaled attention score to obtain the attention weight. The attention is the product of the attention weight and value V.
Follow the diagram from input embedding through Q, K, and V, scaled scores, softmax weights, and the final weighted values. section 9.1.1
Where attention sits determines what information is available. An encoder uses multihead self-attention to relate input tokens. A translation decoder uses masked self-attention for its output sequence and cross-attention to incorporate encoder output. A decoder-only generator such as GPT-2 uses causal self-attention: each token can attend only to itself and earlier tokens. Setting future-token scores to negative infinity before softmax gives those positions zero weight. section 9.1.2 section 11.1.3
Check a causal mask in PyTorch
For a four-token input, split projected Q, K, and V across 25 heads. Their shape becomes [1, 25, 4, 64]: batch, heads, tokens, and values per head. A lower-triangular mask leaves the first token attending only to itself; the fourth can attend to all four. Multiply the masked softmax weights by V, then rejoin the heads to recover shape [1, 4, 1600]. section 11.1.3
The same core technique supports different tasks. An encoder–decoder Transformer uses it for English-to-French translation; a smaller decoder-only music Transformer uses eight attention heads to capture relationships among musical-event tokens and predict subsequent events. section 10.0 section 14.1.2
