U-Net
U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
Build a Text-to-Image Generator (from Scratch)
U-Net is an encoder-decoder neural-network architecture commonly used as the denoising backbone in diffusion models.
- How it is used
- Serves as the noise-predicting backbone for reverse diffusion in image generation, including conditional and latent-space generation.
- How deep it goes
- Traces encoder, bottleneck, decoder, skip connections, training objectives, and attention, progressing from a library model to scratch-built implementations.
- Examples
- Uses Fashion MNIST clothing images, labeled clothing generation, and higher-resolution flower images to demonstrate training and sampling.
- What you can do after
- Build it from scratch as a conditional denoising U-Net, then train and sample with a DDPM; extend the architecture with attention.
Read how Build a Text-to-Image Generator (from Scratch) explains U-Net →
Learn Generative AI with PyTorch
A U-shaped convolutional encoder-decoder neural network with skip connections, used here to predict and remove noise during diffusion-based image generation.
- How it is used
- Uses U-Net as a noise predictor in diffusion-based flower image generation.
- How deep it goes
- Explains encoder, decoder, skip connections, attention, and time embeddings, then walks through model configuration and training.
- Examples
- Uses 64 × 64 Oxford 102 Flower images, a 133-million-parameter U-Net, and a 50-step generation run.
- What you can do after
- Builds and trains a denoising U-Net with PyTorch, then uses it for iterative flower image generation.
Inside Deep Learning: Math, Algorithms, Models
An encoder-decoder neural network architecture for image segmentation that combines low-level and high-level features via skip connections.
- How it is used
- Uses multiscale features to make a prediction for every pixel in an image.
- How deep it goes
- Explains the three subnetworks and implements a reusable PyTorch block that can be nested into a larger architecture.
- Examples
- Implements nested UNetBlock2d modules and compares segmentation results with fine- and coarse-grained models.
- What you can do after
- Builds a segmentation network by nesting UNetBlock2d modules and adding a final convolution for the desired output channels.
Read how Inside Deep Learning: Math, Algorithms, Models explains U-Net →
Local Image Generation
A U-Net is an encoder-decoder neural-network architecture with skip connections for transforming spatial data.
- How it is used
- Uses a neural network to estimate noise during Stable Diffusion image generation.
- How deep it goes
- Explains the architecture’s purpose, learned weights, and training loop, but defers detailed network mechanics.
- Examples
- Uses a U-shaped architecture diagram, Stable Diffusion pipeline, and custom checkpoints for fantasy or photorealistic images.
- What you can do after
- Apply it via Hugging Face Diffusers by loading a pipeline and replacing its U-Net with a task-specific checkpoint.
Grokking AI Algorithms, Second Edition
A U-Net is an encoder-decoder neural-network architecture with skip connections commonly used for image segmentation and image generation.
- How it is used
- Deploys an image-to-image CNN architecture for reconstructing images and powering diffusion-based image generation.
- How deep it goes
- Explains the encoder, bottleneck, decoder, skip connections, denoising objective, and reverse-diffusion use without deep mathematical detail.
- Examples
- Uses an 8×8 noisy image, architecture diagrams, and stepwise denoising from pure noise to a generated image.
- What you can do after
- Build it from scratch as the denoising network in a diffusion model and follow its training and generation workflow.
Read how Grokking AI Algorithms, Second Edition explains U-Net →
Build AI Drug Discovery Pipelines
U-Net is an encoder-decoder neural-network architecture with skip connections that combines representations at multiple spatial scales.
- How it is used
- Uses multiscale sequence features to predict protein residue contacts for constructing graph representations.
- How deep it goes
- Explains encoder-decoder flow, skip connections, feature inputs, and thresholded outputs, but defers layer-level implementation.
- Examples
- Uses protein contact prediction, 72 MSA-derived features, contact probabilities, and a 0.5 graph-construction threshold.
- What you can do after
- Apply U-Net contact predictions to construct protein graphs whose nodes are residues and edges represent predicted contacts.
The AI Pocket Book
An encoder-decoder neural network architecture that reconstructs a full-size output image from a compressed representation, often using intermediate features from the shrinking path.
- How it is used
- Transforms images into full-size reconstructed or stylistically altered images, including caption-guided diffusion outputs.
- How deep it goes
- Explains the shrinking path, abstract representation, intermediate features, and reconstruction, but does not detail layer internals.
- Examples
- Uses damaged-image reconstruction, Van Gogh-style transformation, and caption-guided repair from Gaussian noise.
- What you can do after
- Understand how U-Net-like CNNs reconstruct images and guide iterative text-to-image diffusion.
Also covered in
- Generative AI in Action
- Deep Learning with Python, Third Edition
- Deep Learning with R, Third Edition
What to know first
- convolutional layerA convolutional layer is a neural-network layer that applies shared learned filters to local regions of structured input to produce feature representations.
- convolutional neural networkA convolutional neural network is a neural-network architecture that processes structured inputs with convolutional operations, typically using learned local filters to capture patterns.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- diffusion modelA diffusion model is a generative model that learns to reverse a gradual noising process, generating samples through iterative denoising.
Related concepts
- bottleneck layerhas part · A bottleneck layer is a deliberately narrow intermediate neural-network representation that constrains the dimensionality or information capacity of data passing through the model.
- convolutional neural networkkind of · A convolutional neural network is a neural-network architecture that processes structured inputs with convolutional operations, typically using learned local filters to capture patterns.
- neural networkkind of · A neural network is a computational model composed of interconnected, parameterized units or transformations that learn to map inputs to outputs.
- contracting encoder pathhas part · A contracting encoder path is the progressively downsampling branch of an encoder-decoder architecture that extracts hierarchical representations.
- encoderhas part · An encoder is a component or process that transforms input into an encoded or internal representation for subsequent processing, transmission, or decoding.
- expansive decoder pathhas part · An expansive decoder path is the progressively upsampling branch of an encoder-decoder architecture that reconstructs higher-resolution representations.
Questions about U-Net
What is U-Net?
U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
Which Manning books explain U-Net?
Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Inside Deep Learning: Math, Algorithms, Models, Local Image Generation, Grokking AI Algorithms, Second Edition, Build AI Drug Discovery Pipelines and The AI Pocket Book.
What should I know before learning U-Net?
Manning books that teach U-Net usually introduce convolutional layer, convolutional neural network, parameter and diffusion model first.
What is U-Net a kind of?
U-Net is a kind of convolutional neural network. A convolutional neural network is a neural-network architecture that processes structured inputs with convolutional operations, typically using learned local filters to capture patterns.
What are the parts of U-Net?
U-Net includes bottleneck layer, contracting encoder path, encoder and expansive decoder path.
What is U-Net also called?
U-Net is also called UNet.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






