In attention-based translation, a context vector summarizes the source information selected for a decoding step. Unlike a single fixed thought vector, it combines source representations according to attention weights. section 5.3
Figure 5.6 illustrates sample alignments generated by the RNNsearch-50 model. The plot’s x- and y-axes correspond to words in the source (English) and target (French) sentences, respectively. Pixels indicate the weight assigned by the attention mechanism to each source annotation when predicting each target word, visualized in grayscale (0 = black, 1 = white). Each row shows the weights associated with the annotations, indicating which positions in the source sentence were most important when generating each target word. The alignments are largely monotonic, with strong diagonal weights in each matrix. Still, several non-monotonic alignments also appear. Adjectives and nouns are often ordered differently between French and English. For example, the phrase “European Economic Area” is translated as “zone économique européenne.” The model aligns “zone” with “Area,” initially skipping over “European” and “Economic,” then revisits preceding words to complete the full phrase. Used with permission granted by the lead author (Dzmitry Bahdanau).
The alignment view shows how attention weights select source words as each target word is generated; for example, “zone” aligns with “Area” before the model revisits earlier words. section 5.3
