Denoising is the reverse diffusion process: it turns random noise into a clean image through many small steps. Forward diffusion supplies the training task by gradually adding noise to real images. A trained model learns to undo those changes; generation then starts from noise rather than an existing clean image. section 1.3
Trace one diffusion model from training to generation:
- Choose a noise schedule and add noise to clean training images at selected time steps. A linear or cosine schedule changes how quickly image structure disappears. section 5.0 section 5.4
- Train a U-Net on each noisy image and its time step to predict the added noise. Its encoder–decoder structure and skip connections help retain both broad structure and fine detail. section 5.2
- Start generation with random noise. Repeatedly use the trained model to obtain a less noisy sample until a clean image emerges. section 5.2 section 5.4
Why learn the reverse step instead of simply subtracting known noise? Given a noisy image, the preceding clean(er) image is unknown. Training data lets the model approximate that reverse transition. A practical first implementation uses a noise scheduler and a U-Net from diffusers to generate grayscale clothing-item images from Fashion MNIST. section 5.2 section 5.0
For text-to-image generation, the same iteration can be conditioned on a prompt. Latent diffusion denoises a compact image representation instead of every full-resolution pixel, reducing computation; a VAE decoder turns the finished latent image into the output image. This trades work in the diffusion process for a separate reconstruction stage, with possible image-quality costs. section 1.3.2 section 10.2.1
Figure 9.3 How a trained LDM generates an image based on a text prompt. The text prompt shown at the top-left corner is first encoded into a text embedding. To generate an image in the lower-dimensional space (latent space), we start with an image of pure noise (bottom left). We use the trained U-Net to iteratively denoise the image, conditional on the text embedding so the generated image matches the text prompt, with the guidance of a trained CLIP model. The generated latent image (bottom right) is presented to a trained VAE to convert it to a high-resolution image, which is the final output (top right).
Follow the prompt embedding into the U-Net, then the denoised latent image into the VAE decoder; the diagram separates iterative denoising from final image reconstruction. section 9.1.2
