An encoder–decoder model turns one sequence into another. The encoder processes the input, and the decoder uses its representation to produce output tokens. Machine translation makes the job concrete: read a sentence in one language, then generate a sentence in another. section 9.1.2 section 9.2.3
In a recurrent encoder–decoder network, the encoder distills the input into a fixed-length context vector. That bottleneck can lose information from long or complex sentences. Attention improves sequence-to-sequence performance, especially for long sentences, but recurrent processing remains sequential. A Transformer instead stacks attention-based layers; its encoder and decoder both use multi-head self-attention. section 9.1.2 section 9.2.3
To make a Transformer encoder–decoder translate, assemble the pieces around the attention layers:
- Tokenize source and target text separately, then give each side its own embedding layer and positional encoding. PyTorch’s bare `nn.Transformer` does not supply those layers. section 9.2.3
- Prepare source and target padding masks, plus a target mask that prevents the decoder from seeing later positions. Check tensor order: the shown data is batch-first, while the Transformer expects sequence-first inputs. section 9.2.3
- During training, feed the decoder the target sequence without its final token and compare its predictions with the target shifted one position forward. Evaluate on held-out data and save the model with the best validation loss. section 9.2.3
Listing 9.11 TranslationTransformer.forward()
>>> def forward(self, src, tgt): ... src, src_key_padding_mask = self.prepare_src( ... src, self.src_pad_idx) ... tgt, tgt_key_padding_mask, tgt_mask = self.prepare_tgt( ... tgt, self.tgt_pad_idx) ... memory_key_padding_mask = src_key_padding_mask.clone() ... output = super().forward( ... src, tgt, tgt_mask=tgt_mask, ... src_key_padding_mask=src_key_padding_mask, ... tgt_key_padding_mask=tgt_key_padding_mask, ... memory_key_padding_mask=memory_key_padding_mask) ... output = rearrange(output, 'T N E -> N T E') ... return self.linear(output)
The `forward()` implementation shows source and target preparation, padding and target masks, and the final projection of decoder output. section 9.2.3
Inspect a translation
A German test sentence about a mother and her young son produces an English translation beginning “a mother and her little son.” The worked inference also returns attention weights for inspection. Individual examples are useful checks; test loss and BLEU provide broader evaluations of the trained model. section 9.2.3
