A diffusion model generates an image by starting with random noise and repeatedly removing some of it. Training supplies the reverse task: take clean images, add noise at known time steps, and teach a denoising network to predict that noise. Unlike image generation that predicts patches in sequence, diffusion refines a noisy image over many steps. section 1.3 section 1.7
Build a denoising diffusion probabilistic model (DDPM) by connecting four parts:
- Set a noise schedule. At each forward time step, combine an image with normally distributed noise; after enough steps, the image approaches random noise. A schedule controls how quickly this happens. section 5.1.1 section 5.4.1
- Train a U-Net on noisy images and their time steps. Sample a time step for each training image, inject known noise, and minimize mean squared error between that noise and the U-Net’s prediction. For conditional generation, also supply a class label. section 5.2 section 6.3.2
- Sample in reverse. Begin with a random-noise image, use the network’s prediction to take a denoising step, and repeat toward time step zero. Classifier-free guidance combines conditional and unconditional predictions to control label influence. section 1.3.1 section 6.3.2
Generate clothing items
Train on normalized Fashion MNIST clothing images and their ten class labels. A DDPM can sample different clothing categories from noise; changing guidance strength changes how strongly the chosen label steers generation. A basic unconditional version can instead generate clothing-like images without selecting a category. section 5.0 section 6.3.2 section 6.3.3
Diffusion can also run in a compact latent space rather than directly on full-resolution pixels. A text-guided model denoises latents using a prompt, then a variational autoencoder decodes the result into an image. This reduces the work of diffusion, but training a latent diffusion model remains computationally expensive; a pretrained model is a practical option. Text-to-image outputs also face geometric and copyright-related concerns. section 1.7 section 9.1.2 section 1.5
Figure 9.3 How a trained LDM generates an image based on a text prompt. The text prompt shown at the top-left corner is first encoded into a text embedding. To generate an image in the lower-dimensional space (latent space), we start with an image of pure noise (bottom left). We use the trained U-Net to iteratively denoise the image, conditional on the text embedding so the generated image matches the text prompt, with the guidance of a trained CLIP model. The generated latent image (bottom right) is presented to a trained VAE to convert it to a high-resolution image, which is the final output (top right).
Trace the text embedding and noisy latent through iterative U-Net denoising, then follow the finished latent through the VAE to the final image. section 9.1.2
