liveBook · Manning

Multimodal

As explained in Machines that Think

Multimodality is the use or integration of multiple modalities, such as text, images, audio, and video, within a computational system.

Multimodality combines types of information such as text, images, audio, and video in a computational system. For language models, it widens the evidence available beyond text, which records experience indirectly. This can help ground language in perceptual signals when interpreting diagrams, answering questions about images, or reasoning about events in video. section 4.5 section 4.5.1

Different signals are not simply mixed in their raw forms. Modality-specific encoders transform units such as text tokens, image patches, and audio segments into vector representations. These can be mapped into a shared space, where related signals from different modalities are positioned closer together. section 4.5.2

One way to learn that space is contrastive training on paired data: an image and its caption, for example, are brought closer together while unrelated pairs are pushed apart. The resulting alignment can support image retrieval from text, image-guided responses, or generation shaped by audio without first translating it into words. section 4.5.2

Figure 4.18 Learning a shared multimodal representational space through contrastive alignment. Text, images, and audio are mapped into a shared space. Related inputs across modalities are pulled closer together, while unrelated inputs are pushed apart, allowing cross-modal associations to be learned through relative distance.

The figure shows how related text, images, and audio are aligned in a shared space. section 4.5.2

Fusion determines when information from different modalities interacts:

  • Late fusion combines representations near the output, after mostly separate processing. It is relatively simple and stable, but limits earlier cross-modal influence. section 4.5.3
  • Early fusion merges representations soon after encoding and processes them jointly. It enables richer interaction but increases complexity and sensitivity to differences in scale, resolution, or noise. section 4.5.3
  • Cross-attention in a shared Transformer backbone lets one modality selectively query another while retaining modality-specific processing paths. section 4.5.3
Figure 4.19 Early and late fusion strategies in multimodal models. In late fusion, each modality is processed mostly independently, and the representations are combined only near the output. In early fusion, modality-specific encoders produce representations that are merged earlier and then processed jointly by shared layers. The choice of fusion strategy determines how deeply modalities can influence one another during learning, trading simplicity and modularity for richer cross-modal interaction.

The figure contrasts early and late fusion and their effects on cross-modal interaction. section 4.5.3

Broader input does not by itself guarantee understanding. Captions, transcripts, and descriptions help organize perceptual signals in a linguistic frame, while training reinforces recurring associations. A model may associate smoke with fire without establishing that fire causes smoke; perceptual grounding alone is not causal explanation or evidence of learning through intervention. section 4.5.4 section 4.8

How Machines that Think covers it

How deep it goes
Explains shared representations, contrastive alignment, and fusion strategies, while distinguishing perceptual grounding from causal understanding.
Examples
Uses image-caption and audio-transcript pairs, image retrieval, diagram interpretation, and smoke–fire associations to illustrate capabilities and limits.
What you can do after
Evaluate/understand how multimodal alignment and fusion support perceptual tasks without establishing causal understanding.
How it connects
Connects grounding with contrastive learning, attention, Transformers, and causation.

Also explained in

Compare how 13 books explain multimodal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.