CLIP connects visual understanding with natural-language descriptions, allowing image classification from labels supplied at runtime rather than task-specific training. section 6.2.2
Listing 6.12 Using a multimodal model to simultaneously process visual and textual data
from transformers import CLIPProcessor, CLIPModel
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") #1
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["cat", "dog", "tiger", "train"]
inputs = processor(text = labels, #2
images = image,
return_tensors = "pt",
padding=True)
model.eval()
with torch.no_grad():
outputs = model(**inputs) #3
logits_per_image = outputs.logits_per_image #4
probs = logits_per_image.softmax(dim=1) #5
most_likely_index = torch.argmax(probs, dim=1).item() #6
most_likely_object = labels[most_likely_index]
print(f"The most likely object is: {most_likely_object}")
Load the pretrained OpenAI CLIP model and processor, pass an image with candidate labels, then select the highest-probability label. section 6.2.2
The workflow:
- Define candidate text labels such as cat, dog, tiger, and train. section 6.2.2
- Process the labels and image together, run inference without gradients, convert image logits to probabilities, and choose the largest value. section 6.2.2
- For the dog image, the predicted label is dog. section 6.2.2
