To decode is to turn a learned representation into an output. In an encoder–decoder transformer, the encoder represents a source sentence or image; the decoder uses that representation and earlier output tokens to generate a sequence, one token at a time. An image decoder has a different output: a VQGAN or VAE decoder reconstructs an image from image tokens or a compressed latent representation. section 2.1.2 section 1.2.1 section 9.1.1
For sequence decoding, trace the data through three stages:
- Prepare the target input as tokens and embeddings. During training, shift the target sequence so each position learns to predict the next token. A causal mask blocks access to later positions. section 4.1.2 section 2.1.1
- Pass the target representations through decoder layers. Masked self-attention uses available target tokens; cross-attention incorporates the encoder output; a feed-forward sublayer further transforms the result. The implemented layers also use residual connections. section 2.3.2
- Convert the decoder output into token probabilities with a generator. At inference time, append the chosen token and decode again, stopping at an end token. section 2.1.2 section 2.4.2
Listing 2.6 Creating a decoder layer
class DecoderLayer(nn.Module):
def __init__(self, size, self_attn, src_attn,
feed_forward, dropout):
super().__init__()
self.size = size
self.self_attn = self_attn
self.src_attn = src_attn
self.feed_forward = feed_forward
self.sublayer = nn.ModuleList([deepcopy(
SublayerConnection(size, dropout)) for i in range(3)])
def forward(self, x, memory, src_mask, tgt_mask):
x = self.sublayer[0](x, lambda x:
self.self_attn(x, x, x, tgt_mask)) #1
x = self.sublayer[1](x, lambda x:
self.src_attn(x, memory, memory, src_mask)) #2
output = self.sublayer[2](x, self.feed_forward) #3
return output
The decoder-layer listing shows the separate calls for masked target self-attention, attention over encoder memory, and the feed-forward network. section 2.3.2
From an image to a caption
An image encoder produces patch-based features. Given the caption input [<start>, two, bears, fight], the decoder is trained against [two, bears, fight, <end>] using cross-entropy loss. When generating a caption, it begins with <start>, repeatedly predicts and appends a token, and stops at <end>. section 4.1.2 section 4.1
Do not confuse that token-by-token loop with decoding a latent image. In latent diffusion, denoising first produces a compact image representation; a pretrained VAE decoder then converts it into a high-resolution image. For example, a 4 × 64 × 64 latent becomes a 3 × 512 × 512 image. Comparing original and reconstructed images helps reveal what compression retained or lost. section 9.3.3 section 9.2.2
