Soft alignment gives each source position a continuous relevance weight for every target-generation step. In RNNsearch, the decoder can dynamically focus on different source annotations instead of compressing the entire sentence into one fixed-length context vector, easing the encoder-decoder information bottleneck. section 5.3
Reordering “European Economic Area”
When translating “European Economic Area” as “zone économique européenne,” the model gives strong weight to “Area” while revisiting earlier words. This non-monotonic pattern lets it handle adjective–noun order differences more flexibly than a strictly left-to-right alignment. section 5.3
Figure 5.6 illustrates sample alignments generated by the RNNsearch-50 model. The plot’s x- and y-axes correspond to words in the source (English) and target (French) sentences, respectively. Pixels indicate the weight assigned by the attention mechanism to each source annotation when predicting each target word, visualized in grayscale (0 = black, 1 = white). Each row shows the weights associated with the annotations, indicating which positions in the source sentence were most important when generating each target word. The alignments are largely monotonic, with strong diagonal weights in each matrix. Still, several non-monotonic alignments also appear. Adjectives and nouns are often ordered differently between French and English. For example, the phrase “European Economic Area” is translated as “zone économique européenne.” The model aligns “zone” with “Area,” initially skipping over “European” and “Economic,” then revisits preceding words to complete the full phrase. Used with permission granted by the lead author (Dzmitry Bahdanau).
Inspect the alignment matrix: each row shows the source-position weights used to predict a target word, revealing both diagonal and reordered attention patterns. section 5.3
