Machine translation automatically converts text from one natural language to another. An English-to-French Transformer treats this as a sequence-to-sequence task: its encoder represents the meaning of an English sentence, and its decoder generates French tokens from those representations. Unlike an RNN, which processes a sequence step by step, a Transformer can process inputs nonsequentially during training and attend to relationships between distant words. section 9.0 section 9.1.2
A practical English-to-French pipeline has four parts:
- Break both languages into subword tokens, build a vocabulary for each, and turn tokens into integer indexes. section 9.1.2 section 10.0
- Convert indexes to word embeddings and add positional encodings so the model has information about token order. section 10.0
- Use self-attention in the encoder to relate input tokens; pass its representations to a decoder that predicts the translation one token at a time. section 9.1.1 section 9.1.2
- Train on paired sentences, then use the trained encoder and decoder to translate a new English phrase. section 10.0
A sentence pair
For “I do not speak French” → “Je ne parle pas français,” subword tokenization can split the English into (i, do, not, speak, fr, ench) and the French into (je, ne, parle, pas, franc, ais). The encoder aims to represent the English meaning so the decoder can produce the French wording. Similar English phrasings, such as “I don’t speak French,” should lead to similar representations and translations. section 9.1.2
Figure 9.4 The Transformer architecture. The encoder in the Transformer (left side of the diagram), which consists of N identical encoder layers, learns the meaning of the input sequence and converts it into vectors that represent its meaning. It then passes these vectors to the decoder (right side of the diagram), which consists of N identical decoder layers. The decoder constructs the output (e.g., the French translation of an English phrase) by predicting one token at a time, based on previous tokens in the sequence and vector representations from the encoder. The generator on the top right is the head attached to the output from the decoder so that the output is the probability distribution over all tokens in the target language (e.g., the French vocabulary).
Follow the encoder’s input representations into the decoder, then the generator’s probability distribution over target-language tokens. The diagram shows why translation needs both an understanding of the source sequence and token-by-token output generation. section 9.1.2
