Downsampling reduces a representation's spatial resolution. In image models, convolutional and pooling layers shrink feature maps while extracting increasingly abstract patterns such as edges, textures, and shapes. section 6.2 section 6.5
Trace downsampling through an encoder–decoder architecture:
- Process the input through convolutional blocks that detect patterns and pooling or strided operations that reduce height and width. section 6.5
- Increase feature depth as spatial dimensions decrease, producing a compact representation with more abstract information. section 5.2
- Pass the smallest, most condensed feature maps through the bottleneck before reconstruction. section 6.5
- Recover spatial resolution with upsampling and preserve fine detail through skip connections from the downsampling path. section 6.5
A 16× latent reduction
A VQGAN encoder applies four strided convolutions, each halving resolution. A 384×384 image therefore becomes a 24×24 latent grid containing 576 feature vectors, each with 256 channels. section 11.5.2
Listing 5.5 Importing a U-Net model from the diffusers library
import diffusers, torch
device="cuda" if torch.cuda.is_available() else "cpu"
model=diffusers.UNet2DModel(sample_size=32, #1
in_channels=1,
out_channels=1,layers_per_block=2,
block_out_channels=(128,128,256,512), #2
down_block_types=("DownBlock2D","DownBlock2D", #3
"AttnDownBlock2D","DownBlock2D",),
up_block_types=("UpBlock2D","AttnUpBlock2D", #4
"UpBlock2D","UpBlock2D",),).to(device)
Inspect a diffusion U-Net configuration that separates downsampling blocks from upsampling blocks and increases feature channels across encoder stages. section 5.2
