A translation model faces a bottleneck if it must compress an entire input sentence into one fixed-length “thought vector” before producing any output. In recurrent encoder–decoder models, the final encoder state can lose details from earlier words, especially in long sentences. Attention offers a different route: at each output step, the decoder can draw on the parts of the input that matter then, rather than rely on one summary made in advance. section 5.3 section 4.4.5
Bahdanau attention made that route concrete for neural machine translation. A bidirectional encoder builds a representation for each source word. As the decoder predicts a target word, it scores the relevance of those source representations, turns the scores into weights, and takes a weighted average to form a context vector. The focus can change for the next prediction. This is attention as a learned alignment between what the decoder needs now and what the encoder has retained. section 5.3
A translation that changes word order
In an English-to-French alignment, “European Economic Area” becomes “zone économique européenne.” When producing “zone,” the model places weight on “Area,” then returns to the preceding English words for the rest of the phrase. Attention need not march through the source in order; its weights can follow the translation’s changing needs. section 5.3
Figure 5.6 illustrates sample alignments generated by the RNNsearch-50 model. The plot’s x- and y-axes correspond to words in the source (English) and target (French) sentences, respectively. Pixels indicate the weight assigned by the attention mechanism to each source annotation when predicting each target word, visualized in grayscale (0 = black, 1 = white). Each row shows the weights associated with the annotations, indicating which positions in the source sentence were most important when generating each target word. The alignments are largely monotonic, with strong diagonal weights in each matrix. Still, several non-monotonic alignments also appear. Adjectives and nouns are often ordered differently between French and English. For example, the phrase “European Economic Area” is translated as “zone économique européenne.” The model aligns “zone” with “Area,” initially skipping over “European” and “Economic,” then revisits preceding words to complete the full phrase. Used with permission granted by the lead author (Dzmitry Bahdanau).
The alignment plots show which English positions receive weight for each French output word, including the reordered phrase. section 5.3
The improvement addressed a practical weakness, not just an architectural preference. For models trained with a 50-word sentence cap, RNNsearch scored 26.75 BLEU against 17.82 for a recurrent encoder–decoder without attention. Scores for the latter fell steeply beyond roughly 20 words, while RNNsearch remained steadier up to 60. Earlier work had shortened the route to useful words by reversing input sentences; attention instead let the decoder access encoder states at each step, regardless of where a word appeared. section 5.3 section 5.7
Table 5.1 BLEU scores for RNN encoder-decoder (RNNencdec) and attention-based (RNNsearch) models across different sentence length caps
|
System
|
Sentence length cap
|
BLEU (All)
|
BLEU (no UNK)
|
|---|---|---|---|
|
RNNencdec30
|
≤30
|
13.93
|
24.19
|
|
RNNsearch30
|
≤30
|
21.50
|
31.44
|
|
RNNencdec50
|
≤50
|
17.82
|
26.71
|
|
RNNsearch50
|
≤50
|
26.75
|
34.16
|
|
RNNsearch50 (longer training)
|
≤50
|
28.45
|
36.15
|
|
Moses (phrasebased)
|
—
|
33.30
|
35.63
|
The BLEU table makes the quality difference between fixed-context and attention-based translation models inspectable across sentence-length caps. section 5.3
The 2017 Transformer took the next step: it replaced recurrent and convolutional layers with an architecture based on attention. In self-attention, positions interact with other positions in the same sequence. A word such as “cheese” can connect directly to “mice” even when many words intervene, rather than waiting for information to pass through a chain of recurrent steps or stacked convolutional layers. Processing need not proceed token by token as it does in a recurrent model, helping training speed. section 5.0 section 5.1.1
Direct access has a cost. If every token interacts with every other token, 500 tokens produce 250,000 pairwise interactions; 1,000 produce one million. Doubling sequence length quadruples that work. Self-attention also faced objections about how it represents order without recurrence, even with positional encodings. Attention removed a fragile information bottleneck, but neither long-context computation nor positional information became free. section 5.1.2
The broader idea is selective access to information. A Neural Turing Machine used attentional heads to read and write an external memory; a Transformer instead lets positions read from the sequence itself. “Attention is all you need” captured the narrower engineering result that recurrence and convolution could be replaced in the Transformer—not a claim that attention alone settles questions about intelligence. section 5.5.1 section 5.8
