A shared representation maps different data modalities into one embedding space, allowing corresponding items to be compared through related vectors. In CLIP, image and text encoders produce modality-specific features, while projection heads convert them into equally sized embeddings. section 8.0 section 8.3.3
Build the shared space by tracing each modality through the model:
- Encode images and captions independently, using an image encoder such as ResNet50 and a text encoder such as DistilBERT. section 8.1.1
- Project the resulting features into a common dimension; the implementation maps 768-value text features and 2,048-value image features to 256-value vectors. section 8.3.3
- Compare every text embedding with every image embedding through a batch similarity matrix, then pull matching pairs together and push mismatched pairs apart. section 8.1.1 section 8.3.3
- Use image-to-image and text-to-text similarities to soften targets, recognizing semantic relationships beyond exact paired matches. section 8.3.3
Selecting an image from a prompt
Given the prompt “a dog walks on the beach,” encode it into the shared space, compare it with thousands of image embeddings, and select the closest image. The same similarity signal can evaluate or guide image generation without generating an image itself. section 8.0
Listing 8.8 Defining a projection head
class ProjectionHead(nn.Module):
def __init__(self,embedding_dim, #1
projection_dim=CFG.projection_dim, #2
dropout=CFG.dropout):
super().__init__()
self.projection = nn.Linear(embedding_dim,
projection_dim)
self.gelu = nn.GELU()
self.fc = nn.Linear(projection_dim, projection_dim)
self.dropout = nn.Dropout(dropout)
self.layer_norm = nn.LayerNorm(projection_dim)
def forward(self, x):
projected = self.projection(x)
x = self.gelu(projected)
x = self.fc(x)
x = self.dropout(x)
x = x + projected
x = self.layer_norm(x) #3
return x
Inspect the projection-head and CLIP-model listings to follow how modality-specific features become comparable embeddings. section 8.3.3
