Multimodal data
Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video.
WordPress Power Toolkit: Harness AI to build next-level websites
Multimodal input is information supplied to an AI system through multiple modalities, such as text, images, audio, or video.
- How it is used
- Explains how AI systems broaden learning and information gathering beyond text to include varied real-world and online sources.
- How deep it goes
- Defines multimodal inputs and illustrates their sources, but stops short of explaining implementation details or model architecture.
- Examples
- Uses traffic cameras, podcasts, YouTube videos, human communications, and current-event web searches as examples.
- What you can do after
- Evaluate how AI systems can draw information from text, images, audio, video, and current web sources.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains multimodal data →
Build AI Drug Discovery Pipelines
A multimodal dataset is a collection combining observations from two or more distinct data modalities.
- How it is used
- Connects diverse drug-discovery data types so insights from one modality can inform analysis of another.
- How deep it goes
- Explains modality representations and dataset harmonization through entity linking, ontology mapping, and relationship confidence scoring.
- Examples
- Uses protein sequences, molecular graphs, cellular images, gene-expression vectors, disease ontologies, and JAK2 identifier matching.
- What you can do after
- Evaluate how heterogeneous biological observations can be represented, aligned, and assigned evidence confidence across data sources.
Read how Build AI Drug Discovery Pipelines explains multimodal data →
Data Analysis with LLMs
Multimodal data combines information from multiple modalities, such as text, images, audio, or video.
- How it is used
- Combines text and image inputs so a language model can answer questions about pictures.
- How deep it goes
- Explains how a message represents text and images as typed elements.
- Examples
- Pairs an image with a banana-presence question and describes generating a poem and picture from instructions.
- What you can do after
- Build a user message containing a text question and image URL for visual question answering.
Also covered in
- Transformers in Action
- Agent Design PatternsIn early access (MEAP)
What to know first
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
Related concepts
- GPT-4oused by · GPT-4o is an OpenAI multimodal generative model in the GPT family that processes and generates text and other supported modalities.
- multimodalused by · A multimodal model is a model that processes, relates, or generates information in more than one data modality.
- promptpart of · A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- outputcontrasts with · An output is a result, value, message, or action produced or returned by a system or operation in response to input or a request.
- conversational aiused by · Conversational AI is artificial intelligence that enables software systems to interpret user input and interact with users through natural-language dialogue.
- type attributeconfigured by · A type attribute is an attribute whose value identifies the kind or format of an element or associated content and influences its interpretation or processing.
Questions about multimodal data
What is multimodal data?
Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video.
Which Manning books explain multimodal data?
WordPress Power Toolkit: Harness AI to build next-level websites, Build AI Drug Discovery Pipelines and Data Analysis with LLMs.
What should I know before learning multimodal data?
Manning books that teach multimodal data usually introduce embedding first.
What is the difference between multimodal data and output?
Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video. An output is a result, value, message, or action produced or returned by a system or operation in response to input or a request.
What is multimodal data part of?
Multimodal data is part of prompt. A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
What is multimodal data also called?
Multimodal data is also called multimodal input.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


