Softmax converts a vector of model outputs into a probability distribution by exponentiating and normalizing its values. The resulting probabilities sum to one, with each value representing a class or category likelihood. section 3.1.5
In a transformer decoder, a final linear transformation followed by softmax produces probabilities for the next token. Neural-network classifiers likewise use it as a final activation. section 3.1 section 3.1.5
Classifying an image with CLIP
A CLIP workflow compares an image with the labels cat, dog, tiger, and train, converts the resulting logits with softmax(), and selects the label with the highest probability. For the demonstrated image, the result is dog. section 6.2.2
Listing 6.12 Using a multimodal model to simultaneously process visual and textual data
from transformers import CLIPProcessor, CLIPModel
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") #1
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["cat", "dog", "tiger", "train"]
inputs = processor(text = labels, #2
images = image,
return_tensors = "pt",
padding=True)
model.eval()
with torch.no_grad():
outputs = model(**inputs) #3
logits_per_image = outputs.logits_per_image #4
probs = logits_per_image.softmax(dim=1) #5
most_likely_index = torch.argmax(probs, dim=1).item() #6
most_likely_object = labels[most_likely_index]
print(f"The most likely object is: {most_likely_object}")
The listing shows the complete CLIP path from processor inputs and model inference to softmax probabilities and the selected label. section 6.2.2
