How deeply does Transformers in Action cover multimodal?

Explains modality encoders and two alignment architectures, then develops API and local inference workflows rather than training an MLLM from scratch.

Multimodal

As explained in Transformers in Action

A multimodal model is a model that processes or generates information across multiple data modalities such as text, audio, video, and images.

A multimodal model works with two or more kinds of data, such as text, images, audio, and video. A multimodal large language model (MLLM) connects these inputs so that information from one can inform its interpretation of another. For example, a product-catalog assistant can use both a written request and a reference image to find a relevant item. section 6.0 section 6.1

The main engineering problem is alignment: pixels, text tokens, waveforms, and video frames do not arrive in the same form. Modality-specific encoders turn inputs into numerical features, which are then aligned for joint processing. An MLLM may also use modality-specific decoders to produce nontext output. section 6.2 section 6.1

Two architectural choices differ in how nontext features reach the language model:

Converter-based alignment maps encoded features into the LLM’s token embedding space. It is comparatively simple and efficient. section 6.2
Perceiver-based alignment makes multimodal tokens available through cross-attention or query-based mechanisms. It permits richer interaction but adds complexity. section 6.2

Table 6.1 Comparison of LLMs and MLLMs

Feature	LLMs	MLLMs
Data processing	Text-only	Multiple modalities with separate encoders
Architecture	Single transformer backbone	Modality encoders + fusion to shared space
Training objective	Next-token prediction	Often adds contrastive cross-modal alignment
Inference cost	Quadratic to sequence length	Text cost plus modality encoding/decoding
Modality encoders	Not used	Encoders for image, audio, video, etc.
Input projection	Text embeddings used directly	Projects modality features into token space
LLM backbone	Processes text sequences	Processes fused multimodal inputs
Output projection	Generates text only	Maps outputs to nontext modalities
Modality generators	None	Generates via modality-specific decoders

Compare text-only LLM components with the encoders, fusion, projection, and optional generators used by MLLMs. section 6.1

Prompt an image model

Encode an image file as a base64 data URL, then send it alongside “What’s in this image?” to Llama 4 Scout through the Novita API. The returned description identifies visible objects and broader scene context, illustrating image analysis driven by a text prompt. section 6.3.2

Ask about video or audio

A locally loaded Qwen2.5-Omni model can answer a question about actions in a video or read text from a screen recording. For audio-only inference, extract a 16 kHz mono waveform, prepare it with the model’s processor, and prompt for musical features, transcription, or a summary. section 6.3.4 section 6.3.6

These outputs can feed multimodal retrieval-augmented generation: extracted information from video can be indexed for later questions, while a pipeline can combine PDFs, images, tables, and text for cross-modal search. section 6.3.4 section 6.5

How Transformers in Action covers it

liveBook manning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Introduction to Generative AI, Second EditionA multimodal interface supports interaction through multiple data modalities, such as text, images, audio, speech, or code.
liveBook manning.com

The Art of AI Product DevelopmentA multimodal model processes or generates information across multiple modalities such as text, images, audio, or video.
liveBook manning.com

LLMs in ProductionA multimodal solution is an AI system or method that processes or combines information from multiple data modalities.
liveBook manning.com

Generative AI in ActionA multimodal model is a model that processes or generates information in multiple modalities such as text, images, audio, or video.
liveBook manning.com

Hugging Face in ActionA multimodal model is a machine learning model that jointly processes or relates inputs from multiple data modalities, such as images and text.
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)A generative model that processes or connects two or more data modalities, such as text, images, audio, or video.
liveBook manning.com

Machine Learning System DesignA property of a system or model that represents or processes information from multiple data modalities, such as text and images.
liveBook manning.com

AI-Powered SearchA generative multimodal model generates content while accepting or representing information from multiple data modalities.
liveBook manning.com

Learn Generative AI with PyTorchA generative model that accepts more than one data modality, such as text plus images or audio, as input.
liveBook manning.com

Build AI-Enhanced Web AppsAI technology that processes and generates content across multiple input or output modalities, such as text, images, and audio.

Compare how 11 books explain multimodal →

Related concepts

language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
CLIPincludes · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
GPT-4includes · GPT-4 is a generative large language model developed by OpenAI for understanding and generating text and other supported inputs and outputs.
data modalityuses · A data modality is a distinct form or channel in which information is represented, conveyed, or processed, such as text, images, or audio.
transformer architectureuses · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.