A probability distribution describes possible values and the probabilities associated with them. In generative models, it represents uncertainty rather than forcing an input to become one fixed value: a VAE encoder produces a distribution in latent space, while a translator's generator produces probabilities over the target vocabulary. section 9.1.1 section 2.3.3
The distribution plays different practical roles across model architectures:
- A VAE encoder parameterizes a normal distribution, typically with a mean and variance, instead of producing one deterministic latent vector. section 9.1.1 section 11.2.2
- Sampling from the latent distribution lets the decoder reconstruct varied outputs and encourages smooth interpolation between data points. section 9.1.1 section 11.2.2
- A transformer generator outputs a distribution over vocabulary tokens and selects the next token autoregressively from prior tokens and encoder output. section 2.3.3
Image encoding
An image encoder can produce a normal distribution in a compact latent space, such as a mean and variance for each latent position. Sampling from it gives the decoder a latent representation from which it reconstructs the image. section 9.1.1
Figure 9.2 How a VAE compresses images to representations in the lower-dimensional (latent) space and then reconstructs the images. A VAE consists of an encoder and a decoder. The encoder compresses images (e.g., with a size of 3 × 512 × 512) into a lower-dimensional space (i.e., the latent space, where images have a size of, e.g., 4 × 64 × 64). The decoder takes the encoded images in the latent space and reconstructs the images to the original high-resolution space (with a size of 3 × 512 × 512 in this example).
Trace the encoder's compression into a 4 × 64 × 64 latent representation and the decoder's reconstruction into a 3 × 512 × 512 image. section 9.1.1
