Multimodal
A multimodal model is a model that processes, generates, or relates information represented in two or more data modalities, such as text, images, audio, or video.
Introduction to Generative AI, Second Edition
A multimodal interface supports interaction through multiple data modalities, such as text, images, audio, speech, or code.
- How it is used
- Enables interactions that cross text, images, audio, or video rather than staying within text alone.
- How deep it goes
- Defines modalities and sketches separate encoders and image–text alignment without implementation-level training instructions.
- Examples
- Uses dog-image descriptions, text-to-image generation, paired captions, GPT-4, and GPT-4o.
- What you can do after
- Evaluate/understand it by distinguishing text-only from multimodal capabilities and recognizing that realistic outputs can remain unreliable.
Read how Introduction to Generative AI, Second Edition explains multimodal →
The Art of AI Product Development
A multimodal model processes or generates information across multiple modalities such as text, images, audio, or video.
- How it is used
- Supports product decisions about combining text with images, audio, or video when richer interactions create user value.
- How deep it goes
- Defines the capability and weighs a product example against infrastructure costs, without detailing model architecture or implementation.
- Examples
- Uses written marketing content and product images to illustrate generated descriptions, social media posts, and suggested image edits.
- What you can do after
- Evaluate/understand when a text-only model is sufficient and when multimodal features warrant greater infrastructure demands.
Read how The Art of AI Product Development explains multimodal →
Transformers in Action
A multimodal model is a model that processes or generates information across multiple data modalities such as text, audio, video, and images.
- How it is used
- Extends language-model reasoning to questions and retrieval involving text, images, audio, or video.
- How deep it goes
- Explains modality encoders and two alignment architectures, then develops API and local inference workflows rather than training an MLLM from scratch.
- Examples
- Uses catalog image search, scene description, video action counting, screen-recording text extraction, and audio analysis.
- What you can do after
- Apply it via Llama 4 Scout’s Novita API or a local Qwen2.5-Omni processor to query images, video, or audio.
LLMs in Production
A multimodal solution is an AI system or method that processes or combines information from multiple data modalities.
- How it is used
- Deploys image-capable language models and frames shared embeddings as a route to cross-modal search.
- How deep it goes
- Defines modalities, explains embedding strategies, and demonstrates LLaVA deployment while deferring comprehensive multimodal treatment.
- Examples
- Uses LLaVA image questions, a model projector, cross-modal animal searches, and audio-processing research as concrete examples.
- What you can do after
- Operate/configure LLaVA with a vision projector and image-query API, while recognizing multimodal dependency risks.
Generative AI in Action
A multimodal model is a model that processes or generates information in multiple modalities such as text, images, audio, or video.
- How it is used
- Handles mixed inputs for enterprise applications such as moderation, recommendation, diagnosis, and situational awareness.
- How deep it goes
- Explains the definition, cross-modal correlation, applications, and stacked transformer-block architecture without mathematical derivation.
- Examples
- Demonstrates image-and-prompt cooking suggestions, alongside moderation, ecommerce, healthcare, and autonomous-driving applications.
- What you can do after
- Apply multimodal prompting through Bing Chat by pairing an image with a task-specific textual request.
Hugging Face in Action
A multimodal model is a machine learning model that jointly processes or relates inputs from multiple data modalities, such as images and text.
- How it is used
- Combines modalities to support tasks such as image captioning, visual question answering, sentiment analysis, and environmental understanding.
- How deep it goes
- Explains modality integration and CLIP’s encoders, alignment, and inference workflow without deriving training mathematics.
- Examples
- Uses CLIP to classify a dog image from the text labels cat, dog, tiger, and train.
- What you can do after
- Apply it via Hugging Face Transformers to classify images using natural-language labels without task-specific fine-tuning.
Build a Text-to-Image Generator (from Scratch)
A generative model that processes or connects two or more data modalities, such as text, images, audio, or video.
- How it is used
- Connects text, images, audio, and video for tasks such as image generation, image editing, and captioning.
- How deep it goes
- Explains modality contrasts and generation pipelines, including patches, denoising, latent spaces, and autoregressive caption decoding.
- Examples
- Uses Stable Diffusion image editing, text-to-image generation, patch prediction, and image captioning from image-caption pairs.
- What you can do after
- Evaluate/understand how multimodal systems map inputs across modalities through transformers or diffusion processes.
Read how Build a Text-to-Image Generator (from Scratch) explains multimodal →
Machine Learning System Design
A property of a system or model that represents or processes information from multiple data modalities, such as text and images.
- How it is used
- Uses intersections of data modalities to enrich datasets during data gathering and support marketing or operational decisions.
- How deep it goes
- Defines modality and multimodal datasets through examples, but stops short of modeling, fusion, or implementation details.
- Examples
- Uses product images with descriptions, satellite imagery with agricultural records, and customer data with behavioral profiles.
- What you can do after
- Apply multimodal data combinations to add missing context and identify opportunities for marketing or operational efficiency.
Read how Machine Learning System Design explains multimodal →
AI-Powered Search
A generative multimodal model generates content while accepting or representing information from multiple data modalities.
- How it is used
- Supports cross-modal retrieval, multimodal ranking, contextual reasoning, and generation within search systems.
- How deep it goes
- Defines multimodal representations and applications, but stops short of detailing training procedures or a complete implementation.
- Examples
- Uses image-to-image search, text-to-image retrieval, image-to-text retrieval, and text-to-image generation as concrete patterns.
- What you can do after
- Apply it via embeddings and cross-modal search to connect queries, documents, images, and generated content.
Learn Generative AI with PyTorch
A generative model that accepts more than one data modality, such as text plus images or audio, as input.
- How it is used
- Extends Transformers beyond text to connect modalities and generate images from language.
- How deep it goes
- Explains inputs, embeddings, contrastive alignment, and iterative denoising, while deferring fuller diffusion and text-to-image treatment.
- Examples
- Uses text-to-image systems, flower-image diffusion, CLIP image–text pairs, and an astronaut-riding-a-unicorn prompt.
- What you can do after
- Understand how models align image and text representations and condition iterative image generation on language.
Read how Learn Generative AI with PyTorch explains multimodal →
Build AI-Enhanced Web Apps
AI technology that processes and generates content across multiple input or output modalities, such as text, images, and audio.
- How it is used
- Extends conversational interfaces by combining text prompts with images for richer, context-aware responses.
- How deep it goes
- Traces image upload, encoding, API forwarding, visual analysis, prompt combination, and response delivery, while deferring exact encoding details.
- Examples
- Uses an uploaded broken-appliance image and a repair question to demonstrate visual identification and contextual analysis.
- What you can do after
- Apply it via an LLM API by sending image data with a text prompt and displaying the generated response.
Related concepts
- language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- CLIPincludes · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
- GPT-4includes · GPT-4 is a generative large language model developed by OpenAI for understanding and generating text and other supported inputs and outputs.
- data modalityuses · A data modality is a distinct form or channel in which information is represented, conveyed, or processed, such as text, images, or audio.
- transformer architectureuses · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.