A [CLS] token is a special element placed at the start of a token sequence. Unlike an image patch or a word, it does not represent one local piece of the input. Its contextual representation serves as a compact summary that a model can use for image classification or a whole-caption embedding. section 3.3.2 section 8.3.1
Trace it through a vision transformer (ViT):
- Divide the image into patches and turn each patch into a vector. For a 32 × 32 CIFAR-10 image, an 8 × 8 patch grid produces 64 patch tokens. section 3.1.2
- Prepend a learnable CLS vector. With 48 values per token, the resulting sequence has shape (65, 48): one CLS token followed by 64 patch tokens. section 3.3.2
- Add learned positional embeddings for all 65 positions, then pass the sequence through the encoder. Send only the encoder output at position zero to the classification layer. section 3.3.2 section 3.3.5
Listing 3.3 Adding positional encoding to the image embedding
class Embeddings(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.patch_embeddings = PatchEmbeddings(config)
self.cls_token = nn.Parameter(torch.randn(1, 1, #1
config.hidden_size))
num_patches = (config.image_size // config.patch_size) ** 2
self.position_embeddings = \ #2
nn.Parameter(torch.randn(1,num_patches+1,config.hidden_size))
def forward(self, x):
x = self.patch_embeddings(x) #3
batch_size, _, _ = x.size()
cls_tokens = self.cls_token.expand(batch_size, -1, -1)
x = torch.cat((cls_tokens, x), dim=1) #4
x = x + self.position_embeddings #5
return x
The PyTorch implementation creates the CLS vector as an nn.Parameter, expands it across a batch, prepends it to the patch tokens, and adds positional embeddings. section 3.3.2
Listing 3.7 Building a classifier based on the ViT
class ViTForClassfication(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.image_size = config.image_size
self.hidden_size = config.hidden_size
self.num_classes = config.num_classes
self.embedding = Embeddings(config)
self.encoder = Encoder(config)
self.classifier = nn.Linear(self.hidden_size,
self.num_classes) #1
self.apply(self._init_weights)
def forward(self, x, output_attentions=False):
embedding_output = self.embedding(x) #2
encoder_output, all_attentions = self.encoder(\ #3
embedding_output,output_attentions=output_attentions)
logits = self.classifier(encoder_output[:, 0, :]) #4
if not output_attentions:
return (logits, None)
else:
return (logits, all_attentions)
The classifier reads encoder_output[:, 0, :], selecting the contextual representation at the CLS position rather than all patch outputs. section 3.3.5
The same sequence-level role appears in caption encoding, but insertion differs. A DistilBERT tokenizer places [CLS] before the caption and [SEP] after it; padding and an attention mask handle unequal caption lengths. A text encoder then takes the final-layer representation at position zero as the caption embedding, suitable for alignment with image embeddings in CLIP. section 8.2.2 section 8.3.1
Listing 8.7 Creating a text encoder
class TextEncoder(nn.Module):
def __init__(self, model_name=CFG.text_encoder_model,
pretrained=CFG.pretrained,
trainable=CFG.trainable):
super().__init__()
if pretrained:
self.model=DistilBertModel.from_pretrained(model_name)
else:
self.model=DistilBertModel(config=DistilBertConfig())
for p in self.model.parameters():
p.requires_grad = trainable
self.target_token_idx = 0 #1
def forward(self, input_ids, attention_mask):
output = self.model(input_ids=input_ids,
attention_mask=attention_mask)
last_hidden_state = output.last_hidden_state #2
return last_hidden_state[:,
self.target_token_idx,:] #3
The text encoder passes token IDs and an attention mask into DistilBERT, then returns last_hidden_state[:, 0, :]. section 8.3.1
Inspect what the image CLS token attends to
For each CIFAR-10 image, select attention values from the CLS position to the 64 patch positions. Averaging the heads and encoder layers yields 64 values that can be reshaped to an 8 × 8 grid and enlarged for comparison with the image. section 3.4.3
