A visual encoder is a neural-network component that converts an image into a numerical image embedding. In CLIP, it produces embeddings with the same dimension as text embeddings, allowing images and descriptions to be compared. section 8.5
A visual encoder participates in two related pipelines:
- CLIP encodes paired images and captions, then uses contrastive learning to increase similarity for matching pairs and decrease it for nonmatching pairs. section 8.5
- Latent diffusion applies the CLIP image encoder to low-resolution latent images, helping align generated images with text embeddings while a U-Net denoises them. section 9.1.2
Image selection by text
After training, compare a text embedding with image embeddings using cosine similarity to select the image in a dataset that best matches a description. section 8.5
Figure 9.3 How a trained LDM generates an image based on a text prompt. The text prompt shown at the top-left corner is first encoded into a text embedding. To generate an image in the lower-dimensional space (latent space), we start with an image of pure noise (bottom left). We use the trained U-Net to iteratively denoise the image, conditional on the text embedding so the generated image matches the text prompt, with the guidance of a trained CLIP model. The generated latent image (bottom right) is presented to a trained VAE to convert it to a high-resolution image, which is the final output (top right).
Trace the encoder's role in text-to-image generation: latent noise is denoised conditionally, then a VAE converts the generated latent image into a high-resolution image. section 9.1.2
