liveBook · Manning

Variational autoencoder

As explained in Deep Learning with Python, Third Edition

A generative latent-variable model that learns an encoder-decoder structure and a probabilistic latent space for reconstructing and sampling data.

A variational autoencoder (VAE) learns a structured latent space that can be sampled to generate new data. Like an autoencoder, it encodes an input and decodes a representation back into a reconstruction. Unlike a classical autoencoder, it encodes an input as parameters of a probability distribution, then samples a latent point to decode. section 17.1.2 section 17.4

Generate MNIST digits

A Keras VAE can encode 28-by-28 MNIST images into a two-dimensional latent space. After training, decode a grid of latent points to see how the generated digits change across that space. section 17.1.3

The model has three main parts:

  • The encoder uses strided convolutions to map an image to two vectors: a mean and a log variance. Strides downsample while retaining information about where features appear. section 17.1.3
  • The sampler draws random noise and combines it with the mean and log variance to produce a latent point. section 17.1.3
  • The decoder reshapes the latent point and uses convolution layers to produce an image with the original dimensions. section 17.1.3
Listing 17.2 Latent space sampling layer
from keras import ops

class Sampler(keras.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.seed_generator = keras.random.SeedGenerator()            #1
        self.built = True

    def call(self, z_mean, z_log_var):
        batch_size = ops.shape(z_mean)[0]
        z_size = ops.shape(z_mean)[1]
        epsilon = keras.random.normal(
            (batch_size, z_size), seed=self.seed_generator            #2
        )                                                              #2
        return z_mean + ops.exp(0.5 * z_log_var) * epsilon            #3

The Keras sampling layer shows how random noise is scaled by the encoded log variance and added to the mean. section 17.1.3

Training is self-supervised: the input images are also the reconstruction targets. The Keras implementation uses a custom compute_loss() method and tracks reconstruction and KL losses. Once trained, the decoder can generate images from chosen latent points without first encoding an input image. section 17.1.3 section 17.4

Listing 17.6 Sampling a grid of points from the 2D latent space and decoding them to images
import matplotlib.pyplot as plt

n = 30                                                                 #1
digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))

grid_x = np.linspace(-1, 1, n)                                        #2
grid_y = np.linspace(-1, 1, n)[::-1]                                   #2

for i, yi in enumerate(grid_y):                                       #3
    for j, xi in enumerate(grid_x):                                    #3
        z_sample = np.array([[xi, yi]])                               #4
        x_decoded = vae.decoder.predict(z_sample)                      #4
        digit = x_decoded[0].reshape(digit_size, digit_size)           #4
        figure[
            i * digit_size : (i + 1) * digit_size,
            j * digit_size : (j + 1) * digit_size,
        ] = digit

plt.figure(figsize=(15, 15))
start_range = digit_size // 2
end_range = n * digit_size + start_range
pixel_range = np.arange(start_range, end_range, digit_size)
sample_range_x = np.round(grid_x, 1)
sample_range_y = np.round(grid_y, 1)
plt.xticks(pixel_range, sample_range_x)
plt.yticks(pixel_range, sample_range_y)
plt.xlabel("z[0]")
plt.ylabel("z[1]")
plt.axis("off")
plt.imshow(figure, cmap="Greys_r")

This grid-sampling example decodes points across the two-dimensional latent space into MNIST images. section 17.1.3

VAEs are useful when reconstruction, interpretable latent spaces, or image editing matters. Their continuous latent spaces can support effects such as changing a frown into a smile or smoothly morphing between images. For high-fidelity image generation, diffusion models are generally the stronger choice. section 17.1.2 section 17.4

How Deep Learning with Python, Third Edition covers it

How deep it goes
Explains the encoder, sampler, decoder, and custom training workflow through a Keras implementation.
Examples
Uses MNIST digits and a decoded two-dimensional latent grid to illustrate generation.
What it assumes
Builds on a previously introduced custom-training workflow while introducing autoencoders.
What you can do after
Builds a digit-generating VAE with Keras and samples its two-dimensional latent space.
How it connects
Connects autoencoders, Bayesian inference, self-supervised learning, latent spaces, and diffusion models.

Also explained in

Compare how 13 books explain variational autoencoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.