A latent space is a compact representation space between an encoder and a decoder. An image encoder maps pixels into a smaller latent representation; a decoder reconstructs an image from it. The representation can retain shapes and object relationships while losing minor pixel-level detail. It is not itself the final high-resolution image. section 9.1.1 section 9.2.2
For a variational autoencoder (VAE), the encoder produces parameters of a distribution over latent space rather than assigning each image one deterministic point. This encourages smoother transitions between representations. By contrast, a basic autoencoder produces deterministic latent vectors and can have difficulty supporting meaningful interpolation or reliable generation of novel images. section 9.1.1 section 11.2.1
The same compact-space idea serves different jobs in image generation:
- Latent diffusion performs its repeated noising and denoising on small image representations instead of full-resolution pixels. A text embedding conditions the denoising U-Net; a VAE decoder turns the resulting latent image into the output image. section 9.1.2 section 10.2.1
- CLIP projects images and captions into a shared embedding space so their vectors can be compared. This supports image–text matching and guidance; it is distinct from the VAE image space that the diffusion model denoises. section 8.0 section 9.1.2
- VQGAN first encodes image patches as continuous latent vectors, then replaces each with a nearby entry from a learned codebook. The resulting integer indices make images available as token sequences for transformers. section 1.2.1 section 11.0
Trace a latent image
Start with a 3 × 512 × 512 image. A pretrained VAE encoder compresses it to 4 × 64 × 64: 48 times fewer values. Decode that latent and compare the reconstruction with the original; the beaver and dolphin examples remain recognizable. For generation, instead start with random noise of shape (1, 4, 64, 64), iteratively denoise it under a text prompt, then decode it. The smaller representation reduces diffusion work, but compression can cost visual detail; the decoder reconstructs the high-resolution output. section 9.2.2 section 9.3.2 section 9.5
Figure 9.2 How a VAE compresses images to representations in the lower-dimensional (latent) space and then reconstructs the images. A VAE consists of an encoder and a decoder. The encoder compresses images (e.g., with a size of 3 × 512 × 512) into a lower-dimensional space (i.e., the latent space, where images have a size of, e.g., 4 × 64 × 64). The decoder takes the encoded images in the latent space and reconstructs the images to the original high-resolution space (with a size of 3 × 512 × 512 in this example).
Follow the encoder from full-resolution pixels into the smaller VAE latent representation, then follow the decoder back to image space. section 9.1.1
