Contrastive learning learns an embedding space by pulling related examples together and pushing nonmatching examples apart. In CLIP, text and images are projected into the same latent space so their semantic similarity can be measured directly. section 8.0 section 8.1
A CLIP-style contrastive-learning pipeline proceeds through these stages:
- Collect a batch of N matching text–image pairs. section 8.1
- Encode each caption and image into embeddings with the same dimension. section 8.1
- Increase similarity for matching pairs while reducing similarity for nonmatching pairs, represented by diagonal and off-diagonal batch relationships. section 8.1.1
- Optimize both retrieval directions with text-to-image and image-to-text cross-entropy losses, then average them into a symmetric objective. section 8.5
Selecting an image from a prompt
After training, encode the prompt “a dog walks on the beach,” compare its embedding with image embeddings from Flickr 8k, and select the image with the highest similarity. section 8.0 section 8.5
Figure 8.2 How the CLIP model is trained
Use the training diagram to trace paired embeddings, matching similarities, and nonmatching similarities across a batch. section 8.1.1
