When predicting a Spanish word from an English sentence, not every source word matters equally. Attention lets the prediction draw directly on the most relevant source tokens. Unlike a recurrent layer, it need not carry information through every intervening position—a useful property when dependencies span long sequences. section 15.3 section 15.3.1
A small dot-product attention function
Given batches of target and source vectors, this R/Keras function scores every source position for every target position, turns each row of scores into weights, and returns weighted source vectors: dot_product_attention <- function(target, source) { scores <- op_einsum("btd,bsd->bts", target, source) scores <- op_softmax(scores, axis = -1) op_einsum("bts,bsd->btd", scores, source) } The softmax makes each target position’s weights sum to one. The output is a context-dependent vector for each target position. section 15.3.1
A learned attention layer first projects its inputs into queries, keys, and values. It compares queries with keys to obtain weights, then uses those weights to combine values. Multi-head attention repeats this in separate projections so a target token can attend to different parts of the source. In Keras, layer_multi_head_attention provides the layer without requiring the scoring code above. section 15.3.1
Figure 15.7 Multiheaded attention allows each target word to attend to different parts of the source sequence in separate partitions of the eventual output vector.
See how separate attention heads can draw on different source positions for the same target word. section 15.3.1
Where attention looks depends on the task:
- Self-attention uses a sequence as its own queries, keys, and values. A word such as “station” can draw context from “train” in the same sentence. section 15.3.2
- Cross-attention lets a translation decoder draw information from an encoder’s source sequence. A source padding mask prevents it from attending to padding. section 15.3.3
- Causal self-attention restricts each decoder position to itself and earlier positions. Without that mask, next-token training could reveal the word the model must predict. section 15.3.3
Attention alone treats tokens as a set: changing their order does not tell it where they occurred. Add positional embeddings to token embeddings when word order matters. In the English-to-Spanish example, adding learned positions raised next-word accuracy from about 58% to 67%. Transformer blocks also pair attention with feedforward layers; attention by itself does not provide their nonlinear transformation. section 15.3.4 section 15.3.5 section 15.3.2
The same selective combination supports classification: a pretrained encoder’s first token can attend to the other positions and supply a context-aware vector for a classification head. During causal text generation, caching past key and value vectors avoids recomputing them for every new token. section 15.4.4 section 16.2.3
