Multimodal
A multimodal model is a model that processes, relates, or generates information in more than one data modality.
Introduction to Generative AI, Second Edition
A multimodal interface supports interaction through multiple data modalities, such as text, images, audio, speech, or code.
- How it is used
- Extends language-based interaction to inputs and outputs across multiple modalities.
- How deep it goes
- Explains separate encoders and contrastive alignment intuitively, without implementation details.
- Examples
- Illustrates image description with a dog, image–caption pairs, and text-to-video generation with a coral reef prompt.
- What you can do after
- Evaluate multimodal capabilities and limitations, including visual inconsistencies and the influence of interfaces on access to information.
Read how Introduction to Generative AI, Second Edition explains multimodal →
The Art of AI Product Development
A multimodal model processes or generates information across multiple modalities such as text, images, audio, or video.
- How it is used
- Supports product discovery by matching data modalities and model capabilities to user needs and team expertise.
- How deep it goes
- Defines multimodal models and compares deployment demands with text-only models without detailing implementation.
- Examples
- Uses a marketing scenario combining written content and product images to produce descriptions, posts, or suggested image edits.
- What you can do after
- Evaluate multimodal options against a text-focused product’s needs and resource demands.
Read how The Art of AI Product Development explains multimodal →
Transformers in Action
A multimodal model is a model that processes or generates information across multiple data modalities such as text, audio, video, and images.
- How it is used
- Extends language-model reasoning to combined text, image, audio, and video inputs.
- How deep it goes
- Explains encoders and converter- versus perceiver-based alignment, then demonstrates image, video, and audio inference workflows.
- Examples
- Uses a product catalog, image description, video action counting, and audio transcription and music analysis.
- What you can do after
- Apply multimodal inference using Llama 4 Scout through the Novita API and Qwen2.5-Omni for video and audio queries.
Understanding AI
A multimodal model is a model that processes or generates information across multiple data modalities such as text, images, audio, or video.
- How it is used
- Supports interactions that combine questions or instructions with images, screenshots, voice, documents, video, or code.
- How deep it goes
- Explains capabilities and links them to tokens, embeddings, attention, and next-token prediction without detailing cross-modal implementation.
- Examples
- Uses image questions, screenshot debugging, image generation, voice-and-text conversation, and early versus modern visual outputs.
- What you can do after
- Evaluate/understand how one model can combine formats while remaining a pattern-prediction system rather than a human-like thinker.
Machines that Think
Multimodality is the use or integration of multiple modalities, such as text, images, audio, and video, within a computational system.
- How it is used
- Motivates multimodal language models as a response to the limits of text-only perceptual grounding.
- How deep it goes
- Explains shared representations, contrastive alignment, and fusion strategies, while distinguishing perceptual grounding from causal understanding.
- Examples
- Uses image-caption and audio-transcript pairs, image retrieval, diagram interpretation, and smoke–fire associations to illustrate capabilities and limits.
- What you can do after
- Evaluate/understand how multimodal alignment and fusion support perceptual tasks without establishing causal understanding.
LLMs in Production
A multimodal solution is an AI system or method that processes or combines information from multiple data modalities.
- How it is used
- Deploys image-capable language models and frames shared embeddings as a route to cross-modal search.
- How deep it goes
- Defines modalities, explains embedding strategies, and demonstrates LLaVA deployment while deferring comprehensive multimodal treatment.
- Examples
- Uses LLaVA image questions, a model projector, cross-modal animal searches, and audio-processing research as concrete examples.
- What you can do after
- Operate/configure LLaVA with a vision projector and image-query API, while recognizing multimodal dependency risks.
Generative AI in Action
A multimodal model is a model that processes or generates information in multiple modalities such as text, images, audio, or video.
- How it is used
- Handles mixed inputs for enterprise applications such as moderation, recommendation, diagnosis, and situational awareness.
- How deep it goes
- Explains the definition, cross-modal correlation, applications, and stacked transformer-block architecture without mathematical derivation.
- Examples
- Demonstrates image-and-prompt cooking suggestions, alongside moderation, ecommerce, healthcare, and autonomous-driving applications.
- What you can do after
- Apply multimodal prompting through Bing Chat by pairing an image with a task-specific textual request.
Hugging Face in Action
A multimodal model is a machine learning model that jointly processes or relates inputs from multiple data modalities, such as images and text.
- How it is used
- Combines modalities to support tasks such as image captioning, visual question answering, sentiment analysis, and environmental understanding.
- How deep it goes
- Explains modality integration and CLIP’s encoders, alignment, and inference workflow without deriving training mathematics.
- Examples
- Uses CLIP to classify a dog image from the text labels cat, dog, tiger, and train.
- What you can do after
- Apply it via Hugging Face Transformers to classify images using natural-language labels without task-specific fine-tuning.
Build a Text-to-Image Generator (from Scratch)
A generative model that processes or connects two or more data modalities, such as text, images, audio, or video.
- How it is used
- Connects text, images, audio, and video for tasks such as image generation, image editing, and captioning.
- How deep it goes
- Explains modality contrasts and generation pipelines, including patches, denoising, latent spaces, and autoregressive caption decoding.
- Examples
- Uses Stable Diffusion image editing, text-to-image generation, patch prediction, and image captioning from image-caption pairs.
- What you can do after
- Evaluate/understand how multimodal systems map inputs across modalities through transformers or diffusion processes.
Read how Build a Text-to-Image Generator (from Scratch) explains multimodal →
Machine Learning System Design
A property of a system or model that represents or processes information from multiple data modalities, such as text and images.
- How it is used
- Uses intersections of data modalities to enrich datasets during data gathering and support marketing or operational decisions.
- How deep it goes
- Defines modality and multimodal datasets through examples, but stops short of modeling, fusion, or implementation details.
- Examples
- Uses product images with descriptions, satellite imagery with agricultural records, and customer data with behavioral profiles.
- What you can do after
- Apply multimodal data combinations to add missing context and identify opportunities for marketing or operational efficiency.
Read how Machine Learning System Design explains multimodal →
AI-Powered Search
A generative multimodal model generates content while accepting or representing information from multiple data modalities.
- How it is used
- Supports cross-modal retrieval, multimodal ranking, contextual reasoning, and generation within search systems.
- How deep it goes
- Defines multimodal representations and applications, but stops short of detailing training procedures or a complete implementation.
- Examples
- Uses image-to-image search, text-to-image retrieval, image-to-text retrieval, and text-to-image generation as concrete patterns.
- What you can do after
- Apply it via embeddings and cross-modal search to connect queries, documents, images, and generated content.
Learn Generative AI with PyTorch
A generative model that accepts more than one data modality, such as text plus images or audio, as input.
- How it is used
- Extends Transformers beyond text to connect modalities and generate images from language.
- How deep it goes
- Explains inputs, embeddings, contrastive alignment, and iterative denoising, while deferring fuller diffusion and text-to-image treatment.
- Examples
- Uses text-to-image systems, flower-image diffusion, CLIP image–text pairs, and an astronaut-riding-a-unicorn prompt.
- What you can do after
- Understand how models align image and text representations and condition iterative image generation on language.
Read how Learn Generative AI with PyTorch explains multimodal →
Build AI-Enhanced Web Apps
AI technology that processes and generates content across multiple input or output modalities, such as text, images, and audio.
- How it is used
- Extends conversational interfaces by combining text prompts with images for richer, context-aware responses.
- How deep it goes
- Traces image upload, encoding, API forwarding, visual analysis, prompt combination, and response delivery, while deferring exact encoding details.
- Examples
- Uses an uploaded broken-appliance image and a repair question to demonstrate visual identification and contextual analysis.
- What you can do after
- Apply it via an LLM API by sending image data with a text prompt and displaying the generated response.
Also covered in
- The AI Pocket Book
- Deep Learning with R, Third Edition
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- Deep Learning with Python, Third Edition
- Build an Advanced RAG Application (From Scratch)In early access (MEAP)
- How Large Language Models Work
- Time Series Forecasting Using Foundation Models
- Deep Learning with PyTorch, Second Edition
- The Complete Obsolete Guide to Generative AI
- Enterprise RAGIn early access (MEAP)
- WordPress Power Toolkit: Harness AI to build next-level websites
- AI Agents and Applications
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
- classifierA classifier is a predictive model that assigns an input to one or more discrete classes or categories.
Related concepts
- language modelkind of · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- CLIPincludes · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
- GPT-4includes · GPT-4 is a multimodal large language model developed by OpenAI that processes text and image inputs and generates text.
- data modalityuses · A data modality is a distinct form or channel in which information is represented or conveyed, such as text, images, audio, or structured data.
- transformer architectureuses · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Questions about multimodal
What is multimodal?
A multimodal model is a model that processes, relates, or generates information in more than one data modality.
Which Manning books explain multimodal?
Introduction to Generative AI, Second Edition, The Art of AI Product Development, Transformers in Action, Understanding AI, Machines that Think, LLMs in Production, Generative AI in Action, Hugging Face in Action and 5 more.
What should I know before learning multimodal?
Manning books that teach multimodal usually introduce language model, embedding, natural language processing and classifier first.
What kinds of multimodal are there?
Kinds of multimodal include CLIP and GPT-4.
What is multimodal a kind of?
Multimodal is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.












