CLIP (contrastive language-image pretraining) learns to place images and their descriptions in a shared embedding space. It measures how well text matches an image; it does not generate an image itself. That makes it useful for image retrieval and for judging or guiding text-to-image generation. section 8.0 section 8.1
Build the matching mechanism in stages:
- Encode captions and images separately. One implementation uses pretrained DistilBERT for text and ResNet50 for images, then projects their 768- and 2,048-value features into 256-value vectors. section 8.1.1 section 8.3.3
- For a batch of 32 matching image–caption pairs, compare every text embedding with every image embedding. The resulting 32 × 32 matrix scores matching pairs on its diagonal and other pairings off it. section 8.3.3
- Train in both directions: distinguish the correct image for each caption and the correct caption for each image. Average the two cross-entropy losses; the shown implementation also derives softer targets from within-modality similarities. section 8.5 section 8.3.3
Listing 8.9 Creating the CLIP model
class CLIPModel(nn.Module):
def __init__(self,temperature=CFG.temperature,
image_embedding=CFG.image_embedding,
text_embedding=CFG.text_embedding):
super().__init__()
self.image_encoder = ImageEncoder()
self.text_encoder = TextEncoder()
self.image_projection = ProjectionHead(
embedding_dim=image_embedding)
self.text_projection = ProjectionHead(
embedding_dim=text_embedding)
self.temperature = temperature
def forward(self, batch):
image_features = self.image_encoder(batch["image"])
text_features = self.text_encoder(
input_ids=batch["input_ids"],
attention_mask=batch["attention_mask"]) #1
image_embeddings = self.image_projection(
image_features)
text_embeddings = self.text_projection(
text_features) #2
logits = \
(text_embeddings @ image_embeddings.T) / self.temperature
images_similarity = image_embeddings @ image_embeddings.T
texts_similarity = text_embeddings @ text_embeddings.T
targets = F.softmax(
(images_similarity +
texts_similarity) / 2 * self.temperature,
dim=-1)
texts_loss = cross_entropy(logits, targets,
reduction='none')
images_loss = cross_entropy(logits.T, targets.T,
reduction='none')
loss = (images_loss + texts_loss) / 2.0 #3
return loss.mean()
The model listing connects the two encoders to separate projection heads, making the path from inputs to comparable embeddings concrete. section 8.3.3
To retrieve an image, encode a prompt, compare it with candidate image embeddings, and rank the scores. Cosine similarity compares vector orientation rather than magnitude; the illustrated matching code ranks dot-product scores. Candidate embeddings can be saved and reused for later prompts. section 8.1.2 section 8.4.2
A prompt need not repeat a caption
A model trained on Flickr 8k selects a classroom image for “students having a class in the classroom,” although its original caption is “a woman helps boys on a computer.” An exercise repeats retrieval with “dogs walk on the beach.” section 8.4.2
A pretrained OpenAI CLIP model provides an alternative to training this smaller model: encode Flickr 8k images once, then display the five highest-scoring matches for a prompt. Training on more image–text pairs gives the pretrained model a more generalizable representation. For generation, CLIP-related text encoding and guidance connect prompts to latent diffusion; classifier-free guidance separately combines conditional and unconditional predictions. section 8.4.3 section 9.1.2 section 9.3.1
