A multimodal model accepts, represents, or generates information across multiple data types, such as text, images, audio, video, and metadata. It extends search beyond text-only matching by connecting different modalities within a shared understanding of content. section 15.1 section 15.6
Use multimodal models to support several search and generation patterns:
- Map images into embeddings for image-to-image search. section 15.1
- Match text queries with images, or use uploaded images to search text. section 15.1
- Generate new content, such as images from text prompts. section 15.1
- Combine textual, audio, video, metadata, sensory, and temporal signals for richer matching, ranking, and reasoning. section 15.6
The underlying pattern is representation: a model learns from multiple input types and encodes their meaning into forms that can support similarity search, classification, question answering, summarization, or generation. Models trained jointly on images and text are also commonly called vision-language models. section 15.1 section 1.3.4
Cross-modal product search
A shopper submits the text query “red hiking jacket,” and the system retrieves relevant product images. The reverse workflow accepts an uploaded image and retrieves matching textual descriptions. Both workflows use relationships learned between the modalities rather than relying only on identical words. section 15.1
Figure 15.1 Foundation model types. LLMs are one of several types of foundation models.
Use the foundation-model taxonomy to distinguish language, vision, and multimodal model families. section 15.1
