Suppose you need a useful image-search baseline before you have enough task-specific data to train a large model. A model trained previously on another dataset can give you a starting point: use its predictions or representations directly, then decide whether adapting it is worth the effort. Large foundation models can also help earlier in the workflow by producing initial labels with a few examples. section 8.4 section 6.2.4
Choose the least demanding form of reuse that fits the task:
- Use predictions directly when the existing task is close enough. An ImageNet-trained classifier, for instance, may already recognize the cats and dogs of interest. section 8.4
- Use representations from a pretrained model to train a small classifier for custom classes. This transfer-learning baseline can be especially effective with a small dataset. section 8.4
- Try a zero-shot or few-shot model or API when it can produce useful results with no or few task-specific training samples. section 8.4
Image search with CLIP
CLIP has text and image encoders trained to associate images with descriptions. For an image-search baseline, their outputs can score a query against an image without new training. A later version could combine query–image and query–description distances, or fine-tune components on the search dataset. section 8.6.2
A pretrained model is not automatically the right production choice. Compare its accuracy, development effort, interpretability, and computation time with simpler baselines. If adaptation is needed, experiments may vary which components to fine-tune, the encoder backbone, prompts, and input preprocessing. section 8.7 section 10.6.2
