Bahdanau attention addresses a central encoder–decoder failure: one fixed-length thought vector loses information from long, semantically rich sequences. It lets the decoder consult different source positions while generating each translated word. section 5.3
In RNNsearch, bidirectional GRUs produce source annotations. At every decoding step, the mechanism scores each annotation, turns scores into weights, and forms a weighted-average context vector. The decoder combines that context with its previous state and last output to predict the next word. section 5.3
Aligning reordered phrases
Attention maps target words to relevant source annotations rather than assuming monotonic translation. In “European Economic Area” → “zone économique européenne,” it first links “zone” with “Area,” then revisits earlier words to complete the phrase. section 5.3
Figure 5.6 illustrates sample alignments generated by the RNNsearch-50 model. The plot’s x- and y-axes correspond to words in the source (English) and target (French) sentences, respectively. Pixels indicate the weight assigned by the attention mechanism to each source annotation when predicting each target word, visualized in grayscale (0 = black, 1 = white). Each row shows the weights associated with the annotations, indicating which positions in the source sentence were most important when generating each target word. The alignments are largely monotonic, with strong diagonal weights in each matrix. Still, several non-monotonic alignments also appear. Adjectives and nouns are often ordered differently between French and English. For example, the phrase “European Economic Area” is translated as “zone économique européenne.” The model aligns “zone” with “Area,” initially skipping over “European” and “Economic,” then revisits preceding words to complete the full phrase. Used with permission granted by the lead author (Dzmitry Bahdanau).
Alignment plots show which source positions influence each generated target word, including diagonal and non-monotonic patterns. section 5.3
