Decoding turns an internal representation into an output. Its job depends on the model: an autoencoder reconstructs an image from a latent vector, while a translation Transformer predicts target-language tokens from an encoded sentence and earlier tokens. Decoding need not recover the input exactly; an autoencoder can lose information during compression. section 7.1.1 section 1.3.2
Reconstruct a handwritten digit
Start with a 28 × 28 grayscale image. A dense-layer encoder compresses its 784 pixel values to a 20-value latent vector; a dense-layer decoder maps that vector back to 784 output values. In the PyTorch implementation, decoder(z) applies a linear layer with ReLU, then another linear layer with sigmoid. Reshape the output to 28 × 28 and compare it with the original. Training updates both halves using a pixel-level reconstruction loss; before training, the reconstructions look like noise. section 7.1.2 section 7.2.2
Figure 7.3 The architecture of an AE and the steps to train one to generate handwritten digits. An AE consists of an encoder (middle left) and a decoder (middle right). In each iteration of training, images of handwritten digits are fed to the encoder (step 1). The encoder compresses the images to deterministic points in the latent space (step 2). The decoder takes the encoded vectors (step 3) from the latent space and reconstructs the images (step 4). The AE adjusts its parameters to minimize the reconstruction loss, the difference between the originals and the reconstructions (step 5).
Follow the latent vector from encoder to decoder, then compare the reconstructed digit with the original to see what the training loss measures. section 7.1.2
For larger color images, a variational autoencoder changes what reaches the decoder: its encoder produces a probability distribution, and a sampled 100-value encoding becomes the decoder input. Linear layers expand that sample; transposed convolutions turn it into a 3 × 256 × 256 face image. Training combines reconstruction loss with KL divergence. Unlike the digit autoencoder’s deterministic latent point, this path decodes a sample. section 7.3.2 section 7.4.1
In an encoder–decoder Transformer, decoding produces a sequence rather than pixels. The decoder uses the encoder’s representation and previously generated target tokens to predict the next token. English-to-French translation starts with BOS and continues until EOS. The PyTorch Transformer keeps source and target embeddings separate and passes the encoder output, source mask, target tokens, and target mask into decode(). section 9.3.2 section 10.0
A denoising U-Net uses “decoder” differently again. Its expansive path upsamples feature maps after a contracting encoder path and bottleneck. In the described diffusion setup, the whole U-Net takes a noisy image and time step and predicts noise; repeated denoising steps reconstruct a clean image. Here, decoding is a path within that prediction model, not a next-token operation. section 15.1.2 section 15.6
