Multimodal data
Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video.
WordPress Power Toolkit: Harness AI to build next-level websites
Multimodal input is information supplied to an AI system through multiple modalities, such as text, images, audio, or video.
- How it is used
- Explains how AI systems broaden learning and information gathering beyond text to include varied real-world and online sources.
- How deep it goes
- Defines multimodal inputs and illustrates their sources, but stops short of explaining implementation details or model architecture.
- Examples
- Uses traffic cameras, podcasts, YouTube videos, human communications, and current-event web searches as examples.
- What you can do after
- Evaluate how AI systems can draw information from text, images, audio, video, and current web sources.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains multimodal data →
Data Analysis with LLMs
Multimodal data combines information from multiple modalities, such as text, images, audio, or video.
- How it is used
- Combines text and image inputs so a language model can answer questions about pictures.
- How deep it goes
- Explains how a message represents text and images as typed elements.
- Examples
- Pairs an image with a banana-presence question and describes generating a poem and picture from instructions.
- What you can do after
- Build a user message containing a text question and image URL for visual question answering.
Related concepts
- GPT-4oused by · GPT-4o is an OpenAI multimodal generative model in the GPT family that processes and generates text and other supported modalities.
- multimodalused by · A multimodal model is a model that processes, generates, or relates information represented in two or more data modalities, such as text, images, audio, or video.
- promptused by · An input, typically containing instructions or context, supplied to an artificial-intelligence model to guide or elicit its output.
- datakind of · Data is represented information, observations, measurements, or other values that can be stored, processed, transmitted, analyzed, or interpreted.
- outputcontrasts with · An output is a result or response produced by a process, operation, or system, including data, content, actions, or signals.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.