How can a model produce an image it has never seen? Instead of choosing pixels directly, it can learn a latent space: a vector space whose points encode statistical structure from training images. A generator, also called a decoder, maps a point back to pixels. Sampling new points can then produce new images. A useful space preserves enough structure that moving through it changes the output in meaningful ways; it is not a model of how the physical world works. section 17.1.1
A two-dimensional space of handwritten digits
A variational autoencoder (VAE) trained on MNIST digits offers a concrete way to see this. Its encoder maps each 28 × 28 image to parameters for a distribution over a two-dimensional latent space. A sampling layer chooses a point, and a decoder turns that point into a digit image. After training, sample a grid of points and decode each one: the resulting grid makes changes across the space visible. section 17.1.3
Listing 17.6 Sampling a grid of points from the 2D latent space and decoding them to images
import matplotlib.pyplot as plt n = 30 #1 digit_size = 28 figure = np.zeros((digit_size * n, digit_size * n)) grid_x = np.linspace(-1, 1, n) #2 grid_y = np.linspace(-1, 1, n)[::-1] #2 for i, yi in enumerate(grid_y): #3 for j, xi in enumerate(grid_x): #3 z_sample = np.array([[xi, yi]]) #4 x_decoded = vae.decoder.predict(z_sample) #4 digit = x_decoded[0].reshape(digit_size, digit_size) #4 figure[ i * digit_size : (i + 1) * digit_size, j * digit_size : (j + 1) * digit_size, ] = digit plt.figure(figsize=(15, 15)) start_range = digit_size // 2 end_range = n * digit_size + start_range pixel_range = np.arange(start_range, end_range, digit_size) sample_range_x = np.round(grid_x, 1) sample_range_y = np.round(grid_y, 1) plt.xticks(pixel_range, sample_range_x) plt.yticks(pixel_range, sample_range_y) plt.xlabel("z[0]") plt.ylabel("z[1]") plt.axis("off") plt.imshow(figure, cmap="Greys_r")
The grid-sampling code shows the practical test: choose coordinates in the learned two-dimensional space, pass each point to the decoder, and arrange the generated digits for inspection. section 17.1.3
The example separates three jobs that are easy to conflate:
- Encode an image. A conventional autoencoder compresses an input and reconstructs it, but that alone does not yield a particularly useful or well-structured space. A VAE instead learns a continuous, structured space suited to generation and editing. section 17.1.2
- Sample a point. The VAE encoder produces a mean and log variance; its sampling layer uses them to choose a latent vector. The decoder maps that vector back to an image. section 17.1.3
- Explore rather than merely reconstruct. Decode fresh points to generate unseen images, or move through a structured space to create edits and smooth transitions. section 17.1.1 section 17.4
“Latent space” also describes internal representations outside an image-generating VAE. In a sentiment classifier, a Dense layer with 16 units projects its input into a 16-dimensional representation space; adding units gives the model more freedom, at greater computational cost and with more opportunity to learn unwanted patterns. In text classification, learned embeddings transform token IDs into a latent space. These representations do different jobs from an image decoder’s input, even though each gives the model a learned space in which to work. section 4.1.3 section 14.6
For image generation, model choice matters. VAEs are useful when a structured, controllable space and reconstruction matter; diffusion models are favored for realistic output and can condition generation on text. In a text-to-image model, prompts first become text embeddings. Interpolating between those embeddings and generating an image at each step reveals a smooth transition between prompts. This is an exploration of the text-conditioning representations, not the same operation as sampling the VAE’s two-dimensional image space. section 17.3.1 section 17.4
Rule of thumb: inspect decoded points or transitions before assuming a space supports the manipulation you want. Plausible images need not be physically coherent; generated hands, lighting, and objects can still go wrong. section 17.1.1 section 17.1.3
