liveBook · Manning

Multimodal data

Also called multimodal input

Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video.

How 3 Manning books explain it

WordPress Power Toolkit: Harness AI to build next-level websites

Multimodal input is information supplied to an AI system through multiple modalities, such as text, images, audio, or video.

How it is used
Explains how AI systems broaden learning and information gathering beyond text to include varied real-world and online sources.
How deep it goes
Defines multimodal inputs and illustrates their sources, but stops short of explaining implementation details or model architecture.
Examples
Uses traffic cameras, podcasts, YouTube videos, human communications, and current-event web searches as examples.
What you can do after
Evaluate how AI systems can draw information from text, images, audio, video, and current web sources.

Read how WordPress Power Toolkit: Harness AI to build next-level websites explains multimodal data →

Build AI Drug Discovery Pipelines

A multimodal dataset is a collection combining observations from two or more distinct data modalities.

How it is used
Connects diverse drug-discovery data types so insights from one modality can inform analysis of another.
How deep it goes
Explains modality representations and dataset harmonization through entity linking, ontology mapping, and relationship confidence scoring.
Examples
Uses protein sequences, molecular graphs, cellular images, gene-expression vectors, disease ontologies, and JAK2 identifier matching.
What you can do after
Evaluate how heterogeneous biological observations can be represented, aligned, and assigned evidence confidence across data sources.

Read how Build AI Drug Discovery Pipelines explains multimodal data →

Data Analysis with LLMs

Multimodal data combines information from multiple modalities, such as text, images, audio, or video.

How it is used
Combines text and image inputs so a language model can answer questions about pictures.
How deep it goes
Explains how a message represents text and images as typed elements.
Examples
Pairs an image with a banana-presence question and describes generating a poem and picture from instructions.
What you can do after
Build a user message containing a text question and image URL for visual question answering.

Read how Data Analysis with LLMs explains multimodal data →

Also covered in

What to know first

Related concepts

Questions about multimodal data

What is multimodal data?

Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video.

Which Manning books explain multimodal data?

WordPress Power Toolkit: Harness AI to build next-level websites, Build AI Drug Discovery Pipelines and Data Analysis with LLMs.

What should I know before learning multimodal data?

Manning books that teach multimodal data usually introduce embedding first.

What is the difference between multimodal data and output?

Multimodal data is information represented across two or more distinct modalities, such as text, images, audio, or video. An output is a result, value, message, or action produced or returned by a system or operation in response to input or a request.

What is multimodal data part of?

Multimodal data is part of prompt. A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.

What is multimodal data also called?

Multimodal data is also called multimodal input.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.