liveBook · Manning

Multimodal

A multimodal model is a model that processes, relates, or generates information in more than one data modality.

How 13 Manning books explain it

Introduction to Generative AI, Second Edition

A multimodal interface supports interaction through multiple data modalities, such as text, images, audio, speech, or code.

How it is used
Extends language-based interaction to inputs and outputs across multiple modalities.
How deep it goes
Explains separate encoders and contrastive alignment intuitively, without implementation details.
Examples
Illustrates image description with a dog, image–caption pairs, and text-to-video generation with a coral reef prompt.
What you can do after
Evaluate multimodal capabilities and limitations, including visual inconsistencies and the influence of interfaces on access to information.

Read how Introduction to Generative AI, Second Edition explains multimodal →

The Art of AI Product Development

A multimodal model processes or generates information across multiple modalities such as text, images, audio, or video.

How it is used
Supports product discovery by matching data modalities and model capabilities to user needs and team expertise.
How deep it goes
Defines multimodal models and compares deployment demands with text-only models without detailing implementation.
Examples
Uses a marketing scenario combining written content and product images to produce descriptions, posts, or suggested image edits.
What you can do after
Evaluate multimodal options against a text-focused product’s needs and resource demands.

Read how The Art of AI Product Development explains multimodal →

Transformers in Action

A multimodal model is a model that processes or generates information across multiple data modalities such as text, audio, video, and images.

How it is used
Extends language-model reasoning to combined text, image, audio, and video inputs.
How deep it goes
Explains encoders and converter- versus perceiver-based alignment, then demonstrates image, video, and audio inference workflows.
Examples
Uses a product catalog, image description, video action counting, and audio transcription and music analysis.
What you can do after
Apply multimodal inference using Llama 4 Scout through the Novita API and Qwen2.5-Omni for video and audio queries.

Read how Transformers in Action explains multimodal →

Understanding AI

A multimodal model is a model that processes or generates information across multiple data modalities such as text, images, audio, or video.

How it is used
Supports interactions that combine questions or instructions with images, screenshots, voice, documents, video, or code.
How deep it goes
Explains capabilities and links them to tokens, embeddings, attention, and next-token prediction without detailing cross-modal implementation.
Examples
Uses image questions, screenshot debugging, image generation, voice-and-text conversation, and early versus modern visual outputs.
What you can do after
Evaluate/understand how one model can combine formats while remaining a pattern-prediction system rather than a human-like thinker.

Read how Understanding AI explains multimodal →

Machines that Think

Multimodality is the use or integration of multiple modalities, such as text, images, audio, and video, within a computational system.

How it is used
Motivates multimodal language models as a response to the limits of text-only perceptual grounding.
How deep it goes
Explains shared representations, contrastive alignment, and fusion strategies, while distinguishing perceptual grounding from causal understanding.
Examples
Uses image-caption and audio-transcript pairs, image retrieval, diagram interpretation, and smoke–fire associations to illustrate capabilities and limits.
What you can do after
Evaluate/understand how multimodal alignment and fusion support perceptual tasks without establishing causal understanding.

Read how Machines that Think explains multimodal →

LLMs in Production

A multimodal solution is an AI system or method that processes or combines information from multiple data modalities.

How it is used
Deploys image-capable language models and frames shared embeddings as a route to cross-modal search.
How deep it goes
Defines modalities, explains embedding strategies, and demonstrates LLaVA deployment while deferring comprehensive multimodal treatment.
Examples
Uses LLaVA image questions, a model projector, cross-modal animal searches, and audio-processing research as concrete examples.
What you can do after
Operate/configure LLaVA with a vision projector and image-query API, while recognizing multimodal dependency risks.

Read how LLMs in Production explains multimodal →

Generative AI in Action

A multimodal model is a model that processes or generates information in multiple modalities such as text, images, audio, or video.

How it is used
Handles mixed inputs for enterprise applications such as moderation, recommendation, diagnosis, and situational awareness.
How deep it goes
Explains the definition, cross-modal correlation, applications, and stacked transformer-block architecture without mathematical derivation.
Examples
Demonstrates image-and-prompt cooking suggestions, alongside moderation, ecommerce, healthcare, and autonomous-driving applications.
What you can do after
Apply multimodal prompting through Bing Chat by pairing an image with a task-specific textual request.

Read how Generative AI in Action explains multimodal →

Hugging Face in Action

A multimodal model is a machine learning model that jointly processes or relates inputs from multiple data modalities, such as images and text.

How it is used
Combines modalities to support tasks such as image captioning, visual question answering, sentiment analysis, and environmental understanding.
How deep it goes
Explains modality integration and CLIP’s encoders, alignment, and inference workflow without deriving training mathematics.
Examples
Uses CLIP to classify a dog image from the text labels cat, dog, tiger, and train.
What you can do after
Apply it via Hugging Face Transformers to classify images using natural-language labels without task-specific fine-tuning.

Read how Hugging Face in Action explains multimodal →

Build a Text-to-Image Generator (from Scratch)

A generative model that processes or connects two or more data modalities, such as text, images, audio, or video.

How it is used
Connects text, images, audio, and video for tasks such as image generation, image editing, and captioning.
How deep it goes
Explains modality contrasts and generation pipelines, including patches, denoising, latent spaces, and autoregressive caption decoding.
Examples
Uses Stable Diffusion image editing, text-to-image generation, patch prediction, and image captioning from image-caption pairs.
What you can do after
Evaluate/understand how multimodal systems map inputs across modalities through transformers or diffusion processes.

Read how Build a Text-to-Image Generator (from Scratch) explains multimodal →

Machine Learning System Design

A property of a system or model that represents or processes information from multiple data modalities, such as text and images.

How it is used
Uses intersections of data modalities to enrich datasets during data gathering and support marketing or operational decisions.
How deep it goes
Defines modality and multimodal datasets through examples, but stops short of modeling, fusion, or implementation details.
Examples
Uses product images with descriptions, satellite imagery with agricultural records, and customer data with behavioral profiles.
What you can do after
Apply multimodal data combinations to add missing context and identify opportunities for marketing or operational efficiency.

Read how Machine Learning System Design explains multimodal →

AI-Powered Search

A generative multimodal model generates content while accepting or representing information from multiple data modalities.

How it is used
Supports cross-modal retrieval, multimodal ranking, contextual reasoning, and generation within search systems.
How deep it goes
Defines multimodal representations and applications, but stops short of detailing training procedures or a complete implementation.
Examples
Uses image-to-image search, text-to-image retrieval, image-to-text retrieval, and text-to-image generation as concrete patterns.
What you can do after
Apply it via embeddings and cross-modal search to connect queries, documents, images, and generated content.

Read how AI-Powered Search explains multimodal →

Learn Generative AI with PyTorch

A generative model that accepts more than one data modality, such as text plus images or audio, as input.

How it is used
Extends Transformers beyond text to connect modalities and generate images from language.
How deep it goes
Explains inputs, embeddings, contrastive alignment, and iterative denoising, while deferring fuller diffusion and text-to-image treatment.
Examples
Uses text-to-image systems, flower-image diffusion, CLIP image–text pairs, and an astronaut-riding-a-unicorn prompt.
What you can do after
Understand how models align image and text representations and condition iterative image generation on language.

Read how Learn Generative AI with PyTorch explains multimodal →

Build AI-Enhanced Web Apps

AI technology that processes and generates content across multiple input or output modalities, such as text, images, and audio.

How it is used
Extends conversational interfaces by combining text prompts with images for richer, context-aware responses.
How deep it goes
Traces image upload, encoding, API forwarding, visual analysis, prompt combination, and response delivery, while deferring exact encoding details.
Examples
Uses an uploaded broken-appliance image and a repair question to demonstrate visual identification and contextual analysis.
What you can do after
Apply it via an LLM API by sending image data with a text prompt and displaying the generated response.

Read how Build AI-Enhanced Web Apps explains multimodal →

Also covered in

What to know first

Related concepts

Questions about multimodal

What is multimodal?

A multimodal model is a model that processes, relates, or generates information in more than one data modality.

Which Manning books explain multimodal?

Introduction to Generative AI, Second Edition, The Art of AI Product Development, Transformers in Action, Understanding AI, Machines that Think, LLMs in Production, Generative AI in Action, Hugging Face in Action and 5 more.

What should I know before learning multimodal?

Manning books that teach multimodal usually introduce language model, embedding, natural language processing and classifier first.

What kinds of multimodal are there?

Kinds of multimodal include CLIP and GPT-4.

What is multimodal a kind of?

Multimodal is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.