A single-modal model processes data from one modality, such as text, images, audio, or numerical data. GPT and BERT process text, while ResNet processes images. section 6.2
Single-modal models support focused tasks:
- Process only visual input for object detection and image segmentation. section 6.2.1
- Contrast with multimodal models, which combine inputs such as images and text. section 6.2
Detecting objects with DETR
The facebook/detr-resnet-50 model processes an image containing a dog and cat, returning confidence scores, labels, and bounding boxes. Drawing those boxes produces a labeled visual result. section 6.2.1
Listing 6.10 Using the facebook/detr-resnet-50 model to detect objects in images
from transformers import DetrImageProcessor, DetrForObjectDetection
import torch
image_processor = DetrImageProcessor.from_pretrained(
"facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
inputs = image_processor(images = image, #1
return_tensors = "pt")
model.eval()
with torch.no_grad():
outputs = model(**inputs) #2
target_sizes = torch.tensor([image.size[::-1]]) #3
results = image_processor.post_process_object_detection( #4
outputs,
target_sizes = target_sizes,
threshold = 0.9)[0]
print(results)
Load DETR through Hugging Face Transformers, preprocess the image, run inference with PyTorch, and post-process detections. section 6.2.1
