An encoder–decoder transformer turns an input sequence into an output sequence. The encoder builds representations of the input; the decoder uses those representations and its previously generated output to predict what comes next. In German-to-English translation, the encoder represents a German sentence and the decoder generates its English translation. An encoder-only transformer stops at input representations, while a decoder-only transformer generates from a prompt without a separate encoder. section 2.1.2 section 2.3
Build the path from input to prediction:
- Embed the source and pass it through encoder layers. Each layer combines multi-head self-attention with a feed-forward sublayer; residual connections and layer normalization support training. section 2.3.2
- Embed the output generated so far and pass it through decoder layers. Masked self-attention prevents a position from seeing later output tokens. section 2.3.2
- Use cross-attention to connect the decoder to the encoder’s output, called memory. A feed-forward sublayer completes each decoder layer. section 2.3.2
At generation time, the decoder predicts one token, appends it to the output sequence, and predicts again. Translation stops when it predicts an end-of-sentence token. For image captioning, a vision transformer can instead encode image patches; the decoder starts with <start> and continues until <end>. The encoder’s input changes, but the pattern of encoding a source and generating a target remains. section 2.1.2 section 4.0 section 4.1
Train an image captioner
Pair an image with the caption “two bears fight.” Feed [<start>, two, bears, fight] to the decoder and compare its predictions with [two, bears, fight, <end>] using cross-entropy loss. A causal attention mask ensures each prediction can depend on earlier caption tokens, not later ones; the target sequence is used for loss calculation, not as decoder input. section 4.1.2
Listing 2.5 Defining the Transformer() class
class Transformer(nn.Module):
def __init__(self, encoder, decoder,
src_embed, tgt_embed, generator):
super().__init__()
self.encoder = encoder #1
self.decoder = decoder #2
self.src_embed = src_embed
self.tgt_embed = tgt_embed
self.generator = generator
def encode(self, src, src_mask):
return self.encoder(self.src_embed(src), src_mask)
def decode(self, memory, src_mask, tgt, tgt_mask):
return self.decoder(self.tgt_embed(tgt),
memory, src_mask, tgt_mask)
def forward(self, src, tgt, src_mask, tgt_mask):
memory = self.encode(src, src_mask) #3
output = self.decode(memory, src_mask,
tgt, tgt_mask) #4
return output
The Transformer class makes the data flow explicit: encode the source into memory, then decode the target input against that memory. section 2.3.2
