An encoder turns an input into a representation that another component can use. What it preserves depends on the task: a translation transformer represents a source sentence for a decoder, while an image encoder might produce patch features, a single embedding, or a compressed latent image. An encoder does not, by itself, generate the translated sentence or image caption. section 2.1.2 section 4.1.2 section 9.1.1
To build a transformer encoder for German-to-English translation, trace the input through these stages:
- Tokenize the German sentence. Convert tokens to word embeddings and add positional encodings so the model receives both token information and order. section 2.1.1
- Pass the embeddings through a stack of encoder layers. Each layer uses multi-head self-attention to relate positions in the input, followed by a feed-forward network; residual connections and layer normalization support training. section 2.1.2 section 2.3.2
- Give the resulting tensor—called memory in the Transformer class—to the decoder. The decoder attends to it while predicting target tokens one at a time. section 2.3.2 section 2.4.2
Listing 2.5 Defining the Transformer() class
class Transformer(nn.Module):
def __init__(self, encoder, decoder,
src_embed, tgt_embed, generator):
super().__init__()
self.encoder = encoder #1
self.decoder = decoder #2
self.src_embed = src_embed
self.tgt_embed = tgt_embed
self.generator = generator
def encode(self, src, src_mask):
return self.encoder(self.src_embed(src), src_mask)
def decode(self, memory, src_mask, tgt, tgt_mask):
return self.decoder(self.tgt_embed(tgt),
memory, src_mask, tgt_mask)
def forward(self, src, tgt, src_mask, tgt_mask):
memory = self.encode(src, src_mask) #3
output = self.decode(memory, src_mask,
tgt, tgt_mask) #4
return output
Follow encode(src, src_mask) into memory, then follow that same memory into decode(...). This is the encoder–decoder handoff in code. section 2.3.2
From source sentence to translation
For “Wie geht es dir,” the encoder processes the German tokens into a meaning-bearing tensor. Given that tensor and previously generated English tokens, the decoder produces “How are you.” At inference time, decoding starts with BOS and stops when it predicts EOS. section 2.1.2 section 2.4.2
The same encoder–decoder pattern works for image captioning. Divide an image into patches, embed them, add positional encodings, and pass them through encoder layers. A caption decoder uses the resulting image features alongside earlier caption tokens to predict the next word. Unlike the translation example, the encoder’s input is visual rather than textual. section 4.1.2 section 4.1
Not every encoder feeds a token-generating decoder. In CLIP, separate text and image encoders produce embeddings in a shared space. Training pulls matching image–text pairs together and separates nonmatching pairs; afterward, embedding similarity can rank images for a prompt. One image-encoder implementation uses pretrained ResNet50 to produce a 2,048-value vector, then projects it to a 256-value embedding. section 8.1.1 section 8.1.2 section 8.3.2
For compression, a VAE encoder maps an image to parameters of a latent distribution; a decoder reconstructs an image from a sampled latent. A pretrained example compresses a 3 × 512 × 512 image to 4 × 64 × 64, then reconstructs it. The smaller representation retains important features but can discard fine detail. VQGAN takes a different route: its image encoder produces patch latents that are matched to a discrete codebook, yielding integer indices a transformer can process. section 9.1.1 section 9.2.2 section 1.2.1
