Chapter 10: Diffusion models for images
The introduction contrasts image generation with text generation while identifying their shared goal of learning data patterns to produce plausible new samples.
After this chapter you will be able to
- Image and text structure. Distinguish text generation’s ordered tokens from image generation’s pixels and spatial relationships, and explain why predicting the next token does not intuitively transfer to images.
- Shared generative modeling goal. Explain that both text and image generation aim to learn the training data’s probability distribution so a model can produce new, plausible samples resembling that data.
- VAE encoding and generation. Trace how a VAE encodes an image into a lower-dimensional latent representation, decodes it to reconstruct the image, and uses the decoder alone with noise to generate a new image.
- GAN adversarial generation. Explain the roles of a GAN’s generator and discriminator during training and identify which component produces new samples after training.
- Practical text-to-image generation. Explain that DALL-E demonstrated diffusion models’ practicality by generating images from text descriptions after training on images paired with captions, and that open-source variants such as Stable Diffusion can generate high-resolution images.
- Controlled forward diffusion. Describe how the forward diffusion process adds noise over successive steps, slowly and predictably degrading an image’s structure.
- Learned reverse diffusion. Explain how a diffusion model learns to counteract noise step by step, turning apparent randomness back into coherent image structure.
- Visualizable 2D point dataset. Describe why the example uses 2D points and how it prepares them: extract an image outline as (x, y) floating-point coordinates, convert the points to a tensor, and plot them.
- Progressive destruction of point data. Explain that forward diffusion repeatedly changes the points until their original arrangement is unrecognizable, with the points becoming approximately normally distributed around zero.
- Per-step noise and scaling. Describe how one forward step scales down the previous points and adds normally distributed random noise, with beta controlling the balance. Explain that the example uses a linearly increasing beta schedule.
- Direct sampling at a timestep. Explain that a closed-form equation uses the original data and a product of alpha values to obtain the diffused data at a chosen timestep, avoiding a loop through earlier steps. State that the implemented function returns both the diffused data and the noise added.
- Noise Prediction Training Objective. Explain that the network receives noisy data and a timestep and learns to predict the noise added, so its predictions can be used to reverse diffusion step by step.
- Embedded Denoising Model Inputs. Describe how the proposed feedforward model embeds the timestep and 2D point coordinates, processes them through linear layers with ReLU activations, and outputs predicted noise.
- Noise prediction loss. Explain that the model is trained by minimizing mean squared error between its predicted noise and the noise actually added to an image, and why accurate predictions matter for generation.
- Noise prediction training loop. Describe how each training iteration selects a random timestep, computes the noise-prediction loss, backpropagates it, and updates the model parameters with an optimizer.
- Iterative reverse diffusion. Explain why generating data with the trained model requires repeated sampling, starting at the last timestep and working backward.
- Noise prediction during reversal. Describe how the learned denoising model helps estimate the preceding timestep’s data, including why a little noise is added back.
- Gradual destruction through noise. Describe the forward diffusion process and its effect on the chapter’s example data.
- Noise prediction and iterative sampling. Explain how the chapter’s model was trained to predict noise and how repeated sampling produced a shape from random points.
What it builds on
- A motivating example: Generating names character by character (chapter 9) · Introduces generating text one element at a time.
- Tensors: Multidimensional arrays (chapter 3) · The reader has encountered PyTorch tensors as data containers.
- Embeddings and linear layers (chapter 9) · The earlier use of embeddings to transform model inputs.
Section by section
Introduction
The introduction contrasts image generation with text generation while identifying their shared goal of learning data patterns to produce plausible new samples.
- Image and text structure: Distinguish text generation’s ordered tokens from image generation’s pixels and spatial relationships, and explain why predicting the next token does not intuitively transfer to images.
Why it matters: This distinction explains why the chapter turns to an approach designed for generating images. - Shared generative modeling goal: Explain that both text and image generation aim to learn the training data’s probability distribution so a model can produce new, plausible samples resembling that data.
Why it matters: It connects the earlier text-generation work to this chapter’s image-generation task.
Key terms: pixels, tokens, spatial dependencies, generative AI, probability distribution, training dataset, samples
10.1 History of VAEs and GANs
This section explains how VAEs and GANs generate images and places their popularity in the history leading up to diffusion models.
- VAE encoding and generation: Trace how a VAE encodes an image into a lower-dimensional latent representation, decodes it to reconstruct the image, and uses the decoder alone with noise to generate a new image.
Why it matters: The encoder–decoder distinction explains why generating a new image needs only the decoder. - GAN adversarial generation: Explain the roles of a GAN’s generator and discriminator during training and identify which component produces new samples after training.
Why it matters: The competition between the two components is what distinguishes GAN training from the VAE approach described here. - Image generation popularity timeline: Describe why GANs overtook VAEs in popularity and when diffusion models became recognized as a viable alternative to GANs.
Why it matters: This timeline explains why the chapter introduces earlier image generators before turning to diffusion models.
Key terms: VAE, recognition model, encoder, generative model, decoder, latent representation, GAN, generator, discriminator, binary classifier
10.2 Motivator for diffusion models
This section motivates diffusion models with text-to-image examples, from DALL-E’s commercial practicality to high-resolution images made by open-source variants.
- Practical text-to-image generation: Explain that DALL-E demonstrated diffusion models’ practicality by generating images from text descriptions after training on images paired with captions, and that open-source variants such as Stable Diffusion can generate high-resolution images.
Why it matters: These examples establish the image-generation capability the chapter sets out to explain.
Key terms: diffusion models, DALL-E, text descriptions, images and their corresponding captions, Stable Diffusion
10.3 Diffusion in detail
This section explains diffusion models as a controlled process that gradually adds noise to an image and a learned process that reverses it to restore image structure.
- Controlled forward diffusion: Describe how the forward diffusion process adds noise over successive steps, slowly and predictably degrading an image’s structure.
Why it matters: The gradual, controlled degradation is what makes it possible to learn a process that reverses it. - Learned reverse diffusion: Explain how a diffusion model learns to counteract noise step by step, turning apparent randomness back into coherent image structure.
Why it matters: Learning to reverse the loss of structure is what lets a diffusion model generate a coherent image from apparent randomness.
Key terms: forward diffusion process, noise, schedule, reverse diffusion process, reversible by design, structure
10.4 Setting up the data
The section prepares a visualizable dataset of 2D points for demonstrating image diffusion.
- Visualizable 2D point dataset: Describe why the example uses 2D points and how it prepares them: extract an image outline as (x, y) floating-point coordinates, convert the points to a tensor, and plot them.
Why it matters: This gives the later diffusion demonstration a simple dataset whose changes can be seen.
Key terms: 2D points, points_from_img, (x,y) coordinates, tensor, plot points
10.5 The forward process
This section explains how forward diffusion progressively turns a dataset of 2D points into noise, and how to sample its state at any timestep without simulating every preceding step.
- Progressive destruction of point data: Explain that forward diffusion repeatedly changes the points until their original arrangement is unrecognizable, with the points becoming approximately normally distributed around zero.
Why it matters: This is the process the model will later learn to reverse. - Per-step noise and scaling: Describe how one forward step scales down the previous points and adds normally distributed random noise, with beta controlling the balance. Explain that the example uses a linearly increasing beta schedule.
Why it matters: The step equation explains how the original signal fades while noise takes its place. - Direct sampling at a timestep: Explain that a closed-form equation uses the original data and a product of alpha values to obtain the diffused data at a chosen timestep, avoiding a loop through earlier steps. State that the implemented function returns both the diffused data and the noise added.
Why it matters: Direct sampling provides noisy data for a chosen timestep without simulating every earlier one.
Key terms: forward process, T, x0, xt, beta, noise schedule, torch.randn, normal distribution, closed-form equation, alpha values
10.6 Training
This section explains what the denoising network is trained to predict, why that prediction enables image generation, and how a simple network represents its inputs.
- Noise Prediction Training Objective: Explain that the network receives noisy data and a timestep and learns to predict the noise added, so its predictions can be used to reverse diffusion step by step.
Why it matters: It connects the network’s training task to the goal of generating new images. - Embedded Denoising Model Inputs: Describe how the proposed feedforward model embeds the timestep and 2D point coordinates, processes them through linear layers with ReLU activations, and outputs predicted noise.
Why it matters: It identifies what information the denoising model uses and what its output represents.
Key terms: noisy (diffused) data, timestep, predicted noise, SinusoidalEmbedding, sinusoidal positional embedding, coordinates, linear layers, ReLU activations
10.6.1 Loss
This section defines a noise-prediction loss for a diffusion model and describes how to train the model with it.
- Noise prediction loss: Explain that the model is trained by minimizing mean squared error between its predicted noise and the noise actually added to an image, and why accurate predictions matter for generation.
Why it matters: The loss connects the training task—predicting added noise—to the goal of generating images. - Noise prediction training loop: Describe how each training iteration selects a random timestep, computes the noise-prediction loss, backpropagates it, and updates the model parameters with an optimizer.
Why it matters: This loop turns the noise-prediction objective into parameter updates that train the denoising model.
Key terms: mean squared error, predicted noise, actual noise, reverse process, timestep, loss, backpropagate, optimizer, Adam optimizer
10.7 Reversing diffusion (how to sample)
This section explains how a trained diffusion model generates an image by repeatedly denoising random points in reverse timestep order.
- Iterative reverse diffusion: Explain why generating data with the trained model requires repeated sampling, starting at the last timestep and working backward.
Why it matters: It explains both how sampling proceeds and why diffusion image generation can be slow. - Noise prediction during reversal: Describe how the learned denoising model helps estimate the preceding timestep’s data, including why a little noise is added back.
Why it matters: The model does not simply output a finished image: its noise estimate guides each reconstruction step. - Random points to logo: Trace the section’s sampling example from random points to the reconstructed PyTorch logo.
Why it matters: The logo example makes the backward sampling process visible.
Key terms: reverse diffusion process, timestep, sample, denoising model, predicted noise, controlled reintroduction of noise, random 2D points, sample_timestep, PyTorch logo
10.8 Conclusion
The conclusion recaps why diffusion models became prominent and how adding noise, learning to predict it, and sampling over timesteps can generate images.
- Diffusion models and predecessors: Distinguish diffusion models from VAEs and GANs in the chapter’s account of image generation.
Why it matters: This explains the motivation for the chapter’s focus on diffusion models. - Gradual destruction through noise: Describe the forward diffusion process and its effect on the chapter’s example data.
Why it matters: The noising process defines the transformation the model learns to reverse. - Noise prediction and iterative sampling: Explain how the chapter’s model was trained to predict noise and how repeated sampling produced a shape from random points.
Why it matters: This connects the training target to the model’s ability to generate a shape from random points.
Key terms: VAEs, GANs, diffusion models, photorealism, diversity, forward diffusion process, noise, 2D points, PyTorch logo, feedforward neural network
10.9 Exercises
10.10 Summary
Where this is used later
- Our model served by Gradio (chapter 17) · The earlier discussion of sampling images with a diffusion model.
Concepts explained in this chapter
- diffusion modelGenerative models that create outputs by reversing a diffusion process, mentioned here as part of the Hugging Face model collection.
- noiseNoise is variation, content, or data treated as unwanted, irrelevant, or distinct from the signal, pattern, or structure of interest.
- noise scheduleA diffusion-model component that controls the sequence of denoising steps during generation.
- betaPer-step noise schedule coefficients that control how much noise is added during diffusion.
- denoisingThe process of removing noise from a corrupted input, here used iteratively in a diffusion model to reconstruct an image.
- torch.randnA PyTorch function that samples random values from a normal distribution into a tensor.
- embeddingA one-dimensional numeric data structure used here as a mathematical object on which sums and dot products can be computed.
- positional embeddingAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
- image generationThe task of creating new visual content from learned patterns, mentioned here as a preceding example task.
- language modelA language model with enough scale to generate and answer with fluent text from prompts.
- trainingThe optimization phase in which a model learns from examples.
- mean squared errorA loss function that measures the average of squared differences between predicted and target values.
- objective functionLoss computed on training data during fitting.
- generationThe task of producing a token sequence as output, conditioned on preceding input or context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.