A variational autoencoder (VAE) is an encoder–decoder model that reconstructs inputs and generates new images from a learned latent space. An ordinary autoencoder maps each input to a fixed encoding and concentrates on reconstruction. A VAE instead encodes each input as a probability distribution, so it can sample different encodings and decode them into varied outputs. section 7.3 section 7.3.1
For a face image, the encoder produces a 100-value latent representation described by vectors of means and standard deviations. Assuming normal distributions makes those two vectors sufficient to describe where to sample. The decoder turns a sampled vector back into an image. Training combines pixel-level reconstruction loss with KL divergence, which compares the encoder’s output distribution with a standard normal distribution. Reconstruction keeps the output related to its input; the distribution constraint helps make latent encodings useful beyond the training images. section 7.3.1 section 7.3.2
A practical face-image build proceeds in small steps:
- Resize color images to 256 × 256 pixels and load batches of 16. section 7.4.1
- Use convolutional layers to compress each 3 × 256 × 256 image; produce mean and standard-deviation vectors, then sample an encoding. section 7.4.1
- Use linear and transposed-convolutional layers to turn the sampled encoding back into a 3 × 256 × 256 image. section 7.4.1
- Update the encoder and decoder to reduce reconstruction loss plus KL divergence. section 7.3.2
Figure 7.6 The architecture of a VAE and the steps to train one to generate human face images. A VAE consists of an encoder (middle upper left) and a decoder (middle bottom right). In each iteration of training, human face images are fed to the encoder (step 1). The encoder compresses the images to probabilistic points in the latent space (step 2; since we assume normal distributions, each probability point is characterized by a vector of means and a vector of standard deviations). We then sample encodings from the distribution and present them to the decoder. The decoder takes sampled encodings (step 3) and reconstructs images (step 4). The VAE adjusts its parameters to minimize the sum of reconstruction loss and the KL divergence. The KL divergence measures the difference between the encoder’s output and a standard normal distribution.
Follow the face image through encoding, sampling, decoding, and the two-part training loss. section 7.3.2
Check reconstruction, then generation
Pass held-out face images through the trained model and compare their reconstructions with the originals. They resemble the originals but lose some detail. Next, draw latent vectors at random and pass them to the decoder: the resulting faces are not copies of training images. The same latent space supports interpolation between encoded images and encoding arithmetic, including changes involving eyeglasses. section 7.4.3 section 7.0
