Chapter 17: Image generation
The chapter introduces image generation as learning a latent visual space and sampling from it to create new pictures.
After this chapter you will be able to
- Latent spaces for image generation. Explain how the chapter characterizes image generation: learning latent visual spaces and sampling from them to make new pictures interpolated from real ones.
- Scope of image generation techniques. Explain that the techniques discussed are not limited to images: similar models could develop latent spaces of sound or music, but the book focuses on pictures because they have produced the most interesting results so far.
- Latent space image sampling. Explain how a learned low-dimensional latent space and a generator produce new images by mapping sampled points to pixel grids.
- Distribution-based VAE encoding. Trace how a VAE encodes an image as parameters of a latent distribution, samples a point, and decodes that point back toward the input image.
- Continuous latent space for editing. Explain why random sampling around an encoded image encourages nearby latent points to decode to similar images and makes the space suitable for concept-vector manipulation.
- Reconstruction and regularization losses. Distinguish the reconstruction loss from the regularization loss in VAE training, including what the latter encourages in the encoder’s distributions.
- VAE image generation pipeline. Trace how the encoder, sampling layer, and decoder work together to generate MNIST digit images. Describe the encoder’s latent distribution parameters and how the decoder turns a sampled point back into an image.
- Backend agnostic custom loss. Explain why the VAE customizes compute_loss() rather than train_step(), and describe what data its loss uses and returns.
- Training and latent space sampling. Describe how the model is trained without an external loss or target data, then explain what decoding sampled latent points reveals.
- Repeated denoising from pure noise. Trace the section’s central idea from removing a little noise with an autoencoder to repeatedly denoising pure noise until it becomes a new image. Distinguish this reverse process from diffusion, which adds noise.
- Undistorted image preparation. Explain how the extracted flower images become an iterable dataset and why they are cropped to a fixed size rather than distorted by resizing.
- Noise prediction for denoising. Explain how the same model is used at each diffusion denoising step: it receives a noise_rates input indicating how much noise to extract and predicts a noise mask to subtract from the image.
- U-Net stages and widths. Describe the U-Net's downsampling, constant-size middle, and upsampling stages, including how layer widths change as image size shrinks and then grows.
- Diffusion time and denoising. Trace the diffusion process from a pure-noise image toward a nearly pure-signal image, explaining what diffusion time indicates at values 1 and 0.
- Cosine diffusion schedule. Explain how the diffusion schedule relates diffusion time to noise and signal rates, and state the relationship maintained by the cosine schedule.
- Noisy image training pass. Trace what the model’s call() method does to a batch of clean images and what it returns.
- Noise prediction loss. Explain what the training loss compares and why the section places it in compute_loss() rather than a backend-specific training step.
- Visual monitoring of generated images. Explain why image generation is monitored visually during training and describe how the custom callback supports that monitoring.
- Training stabilization choices. Explain how the learning-rate schedule and averaged model weights are used during training, and why the averaged weights help.
- Text-conditioned diffusion process. Explain how a text-to-image diffusion model uses an encoded prompt alongside a noisy image during denoising, and how it can generate an image from pure noise and a new prompt after training.
- Meaningful paths between prompts. Explain why intermediate points between two text-prompt embeddings can be used to generate images that morph between the prompts.
- Spherical interpolation rationale. Justify using spherical linear interpolation rather than a straight-line interpolation between text embeddings.
What it builds on
- Building and training the segmentation model (chapter 11) · The earlier image segmentation example that also used strides for downsampling.
- Using fit() with a custom training loop (chapter 7) · Customizing training behavior while using fit().
- Introduction to ConvNets (chapter 8) · Familiarity with convolutional networks helps identify the U-Net as a kind of ConvNet.
- Residual connections (chapter 9) · The earlier discussion introduces residual connections as an architectural pattern.
Section by section
Introduction
The chapter introduces image generation as learning a latent visual space and sampling from it to create new pictures.
- Latent spaces for image generation: Explain how the chapter characterizes image generation: learning latent visual spaces and sampling from them to make new pictures interpolated from real ones.
Why it matters: This is the central idea behind the image-generation approaches introduced in the chapter.
Key terms: image generation, latent visual spaces, sampling
17.1 Deep learning for image generation
This section explains why the book focuses on images even though the techniques discussed can apply to other media.
- Scope of image generation techniques: Explain that the techniques discussed are not limited to images: similar models could develop latent spaces of sound or music, but the book focuses on pictures because they have produced the most interesting results so far.
Why it matters: It distinguishes the techniques’ broader applicability from the book’s chosen focus.
Key terms: latent spaces, images, sound, music
17.1.1 Sampling from latent spaces of images
Explains how sampling from a learned image latent space generates new images, how text can guide that sampling, and why the results may be visually incoherent.
- Latent space image sampling: Explain how a learned low-dimensional latent space and a generator produce new images by mapping sampled points to pixel grids.
Why it matters: This is the basic idea behind the image-generation approaches discussed in the chapter. - Text guidance and visual combinations: Describe how text-conditioning supports language-guided image generation and how latent-space interpolation permits novel combinations of visual concepts.
Why it matters: It explains how a description can steer generation toward an image that combines familiar visual ideas in a new way. - Limits of image coherence: Explain why generated images can combine striking ideas yet depict physically incoherent details.
Why it matters: It helps distinguish convincing visual combinations from genuine understanding of the scene being depicted.
Key terms: latent space, generator, decoder, sampling, text-conditioning, text-to-image models, language-guided image generation, interpolating, coherence of generated images
17.1.2 Variational autoencoders
This section explains how variational autoencoders turn image reconstruction into a way to learn a continuous latent space suitable for generation and editing.
- Classical autoencoder reconstruction: Describe how a classical image autoencoder encodes an image into a low-dimensional code and decodes it to reconstruct the input, and why its latent space is often less useful than a VAE’s.
Why it matters: The classical autoencoder provides the baseline for understanding what a VAE changes. - Distribution-based VAE encoding: Trace how a VAE encodes an image as parameters of a latent distribution, samples a point, and decodes that point back toward the input image.
Why it matters: Sampling from a distribution is the defining change that makes the VAE’s latent representations useful for generation. - Continuous latent space for editing: Explain why random sampling around an encoded image encourages nearby latent points to decode to similar images and makes the space suitable for concept-vector manipulation.
Why it matters: This connects the VAE’s sampling step to its usefulness for controlled image editing. - Reconstruction and regularization losses: Distinguish the reconstruction loss from the regularization loss in VAE training, including what the latter encourages in the encoder’s distributions.
Why it matters: The two losses explain how training balances recognizable reconstructions with a usable latent space.
Key terms: autoencoder, encoder, decoder, code, latent space, VAE, z_mean, z_log_variance, latent normal distribution, stochasticity
17.1.3 Implementing a VAE with Keras
This section builds a Keras VAE for MNIST digits, explains its custom loss workflow, and uses its decoder to explore the learned latent space.
- VAE image generation pipeline: Trace how the encoder, sampling layer, and decoder work together to generate MNIST digit images. Describe the encoder’s latent distribution parameters and how the decoder turns a sampled point back into an image.
Why it matters: These three parts explain how the VAE connects real images to generated ones. - Location preserving encoder downsampling: Explain why this encoder downsamples with strides rather than max pooling.
Why it matters: The downsampling choice supports an encoding that can be used to reconstruct a valid image. - Backend agnostic custom loss: Explain why the VAE customizes compute_loss() rather than train_step(), and describe what data its loss uses and returns.
Why it matters: It shows how to train this VAE without target labels or backend-specific gradient handling. - Training and latent space sampling: Describe how the model is trained without an external loss or target data, then explain what decoding sampled latent points reveals.
Why it matters: The generated grid makes the continuity and structure of the VAE’s latent space visible.
Key terms: encoder network, sampling layer, decoder network, z_mean, z_log_var, latent space, strides, max pooling, information location, reconstruct
17.2 Diffusion models
This section connects autoencoder denoising to diffusion models, explaining how repeated denoising can generate an image from pure noise.
- Denoising and invented detail: Explain that denoising and super-resolution models produce cleaned-up or higher-resolution images, but details absent from the input are educated guesses rather than recovered information.
Why it matters: A sharper image is not necessarily a more faithful record of its input. - Repeated denoising from pure noise: Trace the section’s central idea from removing a little noise with an autoencoder to repeatedly denoising pure noise until it becomes a new image. Distinguish this reverse process from diffusion, which adds noise.
Why it matters: This explains how denoising can become image generation rather than just image cleanup.
Key terms: denoising, image super-resolution, hallucinating, diffusion models, reverse diffusion models, denoising autoencoder, pure noise
17.2.1 The Oxford Flowers dataset
This section introduces the Oxford Flowers dataset and explains how to load and resize its images for image generation without distorting them.
- Oxford Flowers dataset contents: Describe the Oxford Flowers dataset as a collection of 8,189 flower images representing 102 species.
Why it matters: This identifies the images used in the diffusion-model example. - Undistorted image preparation: Explain how the extracted flower images become an iterable dataset and why they are cropped to a fixed size rather than distorted by resizing.
Why it matters: Keeping a fixed image size without stretching the flowers protects the quality of generated images.
Key terms: Oxford Flowers dataset, image_dataset_from_directory(), crop_to_aspect_ratio, aspect ratio
17.2.2 A U-Net denoising autoencoder
This section describes a U-Net denoising model that predicts noise for repeated diffusion steps and uses matching downsampling and upsampling stages to preserve image detail.
- Noise prediction for denoising: Explain how the same model is used at each diffusion denoising step: it receives a noise_rates input indicating how much noise to extract and predicts a noise mask to subtract from the image.
Why it matters: This explains what the model predicts and how that prediction denoises an image. - U-Net stages and widths: Describe the U-Net's downsampling, constant-size middle, and upsampling stages, including how layer widths change as image size shrinks and then grows.
Why it matters: The three stages give a clear picture of the denoising model's overall architecture. - Matching blocks and detail connections: Explain how corresponding downsampling and upsampling blocks are connected and why those connections are included.
Why it matters: These connections explain how the architecture helps preserve details despite reducing spatial size.
Key terms: noise_rates, predicted noise mask, diffusion denoising process, U-Net, downsampling stage, middle stage, upsampling stage, widths, 1:1 mapping, concatenative residual connections
17.2.3 The concepts of diffusion time and diffusion schedule
Defines diffusion time for iterative image denoising and explains how a cosine diffusion schedule maps that time to noise and signal rates.
- Diffusion time and denoising: Trace the diffusion process from a pure-noise image toward a nearly pure-signal image, explaining what diffusion time indicates at values 1 and 0.
Why it matters: Diffusion time tells you where an image is in the progression from noise to signal. - Cosine diffusion schedule: Explain how the diffusion schedule relates diffusion time to noise and signal rates, and state the relationship maintained by the cosine schedule.
Why it matters: The schedule supplies the noise and signal rates that guide denoising at each diffusion time.
Key terms: diffusion process, denoising autoencoder, diffusion time, diffusion schedule, cosine schedule, diffusion_schedule(), diffusion_times, noise_rates, signal_rates
17.2.4 The training process
This section describes how a diffusion model prepares noisy images, predicts their noise masks, and trains the denoiser by comparing predicted noise with the noise added.
- Matched image and noise scales: Explain why the training procedure normalizes images before adding noise.
Why it matters: Matching the scales of images and noise makes the noisy inputs used for training sensible. - Noisy image training pass: Trace what the model’s call() method does to a batch of clean images and what it returns.
Why it matters: This pass creates the noisy examples and the outputs needed to train the denoiser. - Noise prediction loss: Explain what the training loss compares and why the section places it in compute_loss() rather than a backend-specific training step.
Why it matters: The loss makes clear that training targets the added noise, not a direct comparison between clean and denoised images.
Key terms: image normalization layer, unit variance, zero mean, call(), diffusion times, diffusion schedule, noise rates, signal rates, noise masks, mean absolute error
17.2.5 The generation process
17.2.6 Visualizing results with a custom callback
The section explains why generated images should be inspected during training and how a custom callback can display them after each epoch.
- Visual monitoring of generated images: Explain why image generation is monitored visually during training and describe how the custom callback supports that monitoring.
Why it matters: Seeing generated images after each epoch helps you judge whether the model is making progress.
Key terms: custom callback, generate() method, epoch
17.2.7 It’s go time!
The section puts the diffusion model into training on Oxford Flowers, explains two training-stabilization choices, and shows how generated images are monitored.
- Training stabilization choices: Explain how the learning-rate schedule and averaged model weights are used during training, and why the averaged weights help.
Why it matters: These choices explain how the section aims to make training steadier and improve generated images. - Training callbacks and results: Describe how the training run uses callbacks to display generated images and save weights, and what the section reports after training.
Why it matters: The callbacks make the training run’s visual progress observable while preserving its weights.
Key terms: AdamW, InverseTimeDecay, exponential moving average of model weights, Polyak averaging, VisualizationCallback, ModelCheckpoint, epochs
17.3 Text-to-image models
This section explains how text conditions a diffusion model to generate images, then introduces pretrained Stable Diffusion, negative prompts, and limits on output quality.
- Text-conditioned diffusion process: Explain how a text-to-image diffusion model uses an encoded prompt alongside a noisy image during denoising, and how it can generate an image from pure noise and a new prompt after training.
Why it matters: This is the central difference between unconditional image diffusion and text-to-image generation. - Negative prompt guidance: Explain how positive and negative prompts can steer the denoiser toward desired image features and away from undesired ones.
Why it matters: A negative prompt provides a way to discourage unwanted content, such as the color blue, without changing the positive description. - Pretrained generation interface: Describe why the example uses pretrained Stable Diffusion and what the KerasHub TextToImage task handles when generating an image.
Why it matters: It explains why the book demonstrates generation with an existing model rather than training one from scratch. - Sources of visual artifacts: Distinguish the section’s two explanations for visual artifacts in generated images: limits of the model’s understanding and the choice of a smaller model.
Why it matters: It helps explain both why artifacts can persist and why a more capable model was not used in the example.
Key terms: text encoder, text_embeddings, denoising model, prompt, image pairs, negative prompt, positive_prompt, negative_prompt, denoiser, Stable Diffusion, KerasHub
17.3.1 Exploring the latent space of a text-to-image model
The section shows how interpolating between text-prompt embeddings can produce a visual morph between generated images, and explains why spherical interpolation is used.
- Meaningful paths between prompts: Explain why intermediate points between two text-prompt embeddings can be used to generate images that morph between the prompts.
Why it matters: It explains why changing an embedding gradually can produce a meaningful visual transition. - Spherical interpolation rationale: Justify using spherical linear interpolation rather than a straight-line interpolation between text embeddings.
Why it matters: The path through embedding space matters because the denoising model needs meaningful points along it. - Three image generation stages: Trace the three stages of the decomposed generate() function, from prompts to renderable images.
Why it matters: Separating these stages makes it possible to interpolate the embeddings before generating each image. - Positive and negative embedding tensors: Describe the four text-embedding tensors passed to the denoising model and why they include more than a final vector.
Why it matters: It clarifies what the section means by the text embeddings used to condition denoising and interpolation.
Key terms: text encoder, text embeddings, smooth, low-dimensional manifold, interpolation, slerp, spherical linear interpolation, text manifold, generate(), denoise, positive prompt
17.4 Summary
Concepts explained in this chapter
- latent spaceAn encoded internal representation in latent space that preserves the structure needed for generation or manipulation.
- encoderThe sequence model component that reads the source sequence and compresses it into a representation for another model component to use.
- diffusion modelA diffusion-based generative model that starts from noise and iteratively denoises it into an image.
- variational autoencoderA generative latent-variable model that learns an encoder-decoder structure and a probabilistic latent space for reconstructing and sampling data.
- decodeThe part of an autoencoder or variational autoencoder that maps a latent representation back into an output space such as images.
- promptA short text description used to condition image generation, including both positive and negative forms in this section.
- autoencoderA neural network trained to reconstruct its input, often by compressing and then decoding it, used here for denoising images.
- samplingThe process of drawing values or examples from a probability distribution, here to generate new content from a learned space.
- denoisingDenoising is the process of removing unwanted noise or corruption from data while preserving meaningful structure.
- image generation modelA model that generates images, here assumed to operate at large scale and potentially use ViT-style architectures.
- noiseAn initial image-like input made of unstructured random values rather than meaningful visual content.
- reconstruction lossA loss that measures how well a model reproduces its input or a corrupted version of it.
- regularizationA method that constrains a model during training to reduce overfitting and improve generalization.
- KerasHubA Keras-hosted library providing pretrained models and high-level tasks such as Stable Diffusion and TextToImage.
- image_dataset_from_directoryA Keras utility function that builds a dataset from image files organized in class-specific subdirectories.
- callbackA user-defined callback that runs automatically during training to perform extra actions such as monitoring outputs.
- objective functionA function that compares predictions with targets and returns a scalar indicating how wrong the model is for a given task.
- embeddingAn ordered numeric representation used here to encode source tokens and target tokens for attention computations.
- tokenizationThe process of converting text into a sequence of discrete units such as words, characters, or punctuation marks.
- residual connectionA residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
- interpolationA program-like behavior produced by a model that interpolates within trained examples rather than synthesizing novel algorithms.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.