U-Net

As explained in Build a Text-to-Image Generator (from Scratch)

U-Net is an encoder-decoder neural-network architecture commonly used as the denoising backbone in diffusion models.

A U-Net is an encoder–decoder neural network with skip connections between layers at matching resolutions. In a diffusion model, it acts as the denoiser: given a noisy image and its time step, it predicts the noise added to that image. A noise scheduler uses that prediction during reverse diffusion, repeatedly refining random noise toward an image. Unlike a simple convolutional network, the U-shaped design combines broad image context with fine spatial detail. section 5.2 section 6.2.2

Trace a noisy image through the network:

• The contracting path applies convolutional layers to extract features and pooling layers to reduce spatial dimensions. As resolution falls, features represent increasingly abstract patterns. section 6.2.2
• The bottleneck processes the most condensed features before reconstruction. section 6.5
• The expansive path upsamples feature maps. Skip connections merge in features from corresponding encoder stages, helping retain details that downsampling might lose. section 6.2.2 section 6.5

Figure 6.4 The components of the denoising U-Net model

Follow the contracting path, bottleneck, and expansive path in the denoising U-Net diagram; note where encoder features rejoin the decoder. section 6.2.2

To train one, sample a time step for each clean image, add a known amount of random noise, and pass the noisy image and time step to the U-Net. Compare its predicted noise with the noise actually added, then update the weights. One clothing-image implementation uses L1 loss; a conditional DDPM implementation uses mean squared error. The target is the injected noise—not a clean image produced in a single pass. section 5.3.1 section 6.3

For controlled generation, embed class labels alongside time steps so the U-Net can denoise for a requested clothing category. For more complex images, a deeper U-Net can add attention in its downsampling and upsampling paths, dynamically emphasizing relevant spatial regions. The high-resolution flower implementation uses scaled dot-product attention over image pixels and has more than 133 million parameters: added capacity comes with substantial model size. section 6.3 section 7.1.1 section 7.3

From clothing items to flowers

A conditional U-Net can be trained on labeled Fashion MNIST images to generate a requested type of clothing item. An unconditional flower U-Net instead embeds time steps without class labels; paired with a DDIM scheduler, it turns distinct initial noise tensors into different flower images over 50 inference steps. section 6.3.3 section 7.3.1 section 7.4.1

A U-Net can also denoise compact image representations rather than pixels. In latent diffusion, it works alongside a text encoder and a variational autoencoder: text conditions iterative denoising in latent space, then the autoencoder decodes the result into an image. Training such a system is computationally expensive, so a pretrained model is used for the text-to-image example. section 9.1.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes

Traces encoder, bottleneck, decoder, skip connections, training objectives, and attention, progressing from a library model to scratch-built implementations.

Examples

Uses Fashion MNIST clothing images, labeled clothing generation, and higher-resolution flower images to demonstrate training and sampling.

What it assumes

Builds on forward and reverse diffusion while introducing convolution, pooling, upsampling, and embedding components before implementation.

What you can do after

Build it from scratch as a conditional denoising U-Net, then train and sample with a DDPM; extend the architecture with attention.

How it connects

Links U-Net to convolutional neural networks, skip connections, DDPM, DDIM, attention, classifier-free guidance, and latent diffusion.

liveBookmanning.com

Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA U-shaped convolutional encoder-decoder neural network with skip connections, used here to predict and remove noise during diffusion-based image generation.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsAn encoder-decoder neural network architecture for image segmentation that combines low-level and high-level features via skip connections.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA U-Net is an encoder-decoder neural-network architecture with skip connections commonly used for image segmentation and image generation.
• 

  liveBookmanning.com
  The AI Pocket BookAn encoder-decoder neural network architecture that reconstructs a full-size output image from a compressed representation, often using intermediate features from the shrinking path.

Compare how 5 books explain U-Net →

Related concepts

• convolutional neural networkkind of · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• bottleneck layerhas part · A bottleneck layer is a deliberately narrow intermediate neural-network representation that constrains the dimensionality or information capacity of data passing through the model.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• neural networkkind of · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
• contracting encoder pathhas part · A contracting encoder path is the progressively downsampling branch of an encoder-decoder architecture that extracts hierarchical representations.
• expansive decoder pathhas part · An expansive decoder path is the progressively upsampling branch of an encoder-decoder architecture that reconstructs higher-resolution representations.

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.