A multimodal model processes and combines multiple information types, such as images, text, audio, video, or structured data. Combining modalities supports richer representations and tasks that require information from more than one source. section 6.2
Common multimodal applications include:
- Generating image captions from visual content. section 6.2
- Answering questions by reasoning across an image and written question. section 6.2
- Combining speech, text, facial expressions, camera data, lidar, and GPS for specialized systems. section 6.2
CLIP connects image understanding with natural-language descriptions. It uses separate text and image encoders, aligns related image-text pairs during training, and can classify images with user-provided labels without task-specific fine-tuning. section 6.2.2
Listing 6.12 Using a multimodal model to simultaneously process visual and textual data
from transformers import CLIPProcessor, CLIPModel
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") #1
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["cat", "dog", "tiger", "train"]
inputs = processor(text = labels, #2
images = image,
return_tensors = "pt",
padding=True)
model.eval()
with torch.no_grad():
outputs = model(**inputs) #3
logits_per_image = outputs.logits_per_image #4
probs = logits_per_image.softmax(dim=1) #5
most_likely_index = torch.argmax(probs, dim=1).item() #6
most_likely_object = labels[most_likely_index]
print(f"The most likely object is: {most_likely_object}")
Load a pretrained CLIP model and processor, provide an image with candidate text labels, then select the label with the highest probability. section 6.2.2
Classifying an image with text labels
Given the labels “cat,” “dog,” “tiger,” and “train” and an image of a dog, the CLIP pipeline returns “dog” as the most likely object. section 6.2.2
