A recurrent model must carry information through successive steps, which can make distant details hard to use. Attention offers another route: for the token being processed, score other tokens by relevance and pull useful information directly into its representation. A Transformer uses attention, rather than recurrent layers, to pass information across a sequence. section 15.3 section 15.6
Choosing a source word while translating
Suppose a decoder is about to predict one Spanish word. It has a vector for the current target position and vectors for every English source word. Attention scores each source vector against the target vector, turns those scores into weights with softmax, then takes a weighted sum of the source vectors. A source word can matter greatly for one prediction and little for the next. section 15.3.1
def dot_product_attention(target, source): scores = np.einsum("btd,bsd->bts", target, source) #1 scores = softmax(scores, axis=-1) return np.einsum("bts,bsd->btd", scores, source) #2 dot_product_attention(target, source)
The NumPy-style function shows both steps for whole sequences: compute a target-by-source score matrix, then use its softmax weights to combine source vectors. section 15.3.1
A dot product supplies the basic relevance score: vectors closer in the embedding space receive higher scores. Learned projections make this more flexible. A query represents what the target position seeks; keys are compared with that query; values supply the information combined according to the resulting weights. Multi-head attention makes several such comparisons in separate vector partitions, allowing a position to draw on different parts of the source. section 15.3.1
A word gains context
In “The train left the station on time,” self-attention can give “station” a high score for “train.” Incorporating information from “train” helps represent which kind of station is meant. Here the query, keys, and values all come from the same sequence. section 15.3.2
Where attention goes in an encoder–decoder Transformer:
- The encoder uses self-attention so each source token can draw on other source tokens. Feedforward layers add the nonlinearity that attention’s linear combinations alone lack. section 15.3.2
- The decoder first attends within the target sequence, then uses cross-attention to draw on encoder output. A source mask excludes padding from cross-attention. section 15.3.3
- Decoder self-attention needs a causal mask: a target position may use itself and earlier positions, but not future words it is supposed to predict. section 15.3.3
[ [1, 0, 0, 0, 0], [1, 1, 0, 0, 0], [1, 1, 1, 0, 0], [1, 1, 1, 1, 0], [1, 1, 1, 1, 1], ]
Read each row of this lower-triangular mask as the positions available to one target token; the permitted positions grow from left to right. section 15.3.3
Attention alone does not know word order: it relates tokens as elements of a set. A translation model built without position information reached about 58% next-word accuracy in the worked run, below its recurrent counterpart. Positional encoding supplies order information when the task requires it. Rule of thumb: use attention for contextual access, but do not mistake it for a complete sequence model by itself. section 15.3.4 section 20.1.6
Attention also supports classification. An encoder’s first token can attend to the rest of a review, making its final representation a contextual alternative to averaging all token vectors before a classification head. Whether translating or classifying, choose the attention pattern and masks to match what each prediction is allowed to see. section 15.4.4 section 15.3.3
