Multimodality combines types of information such as text, images, audio, and video in a computational system. For language models, it widens the evidence available beyond text, which records experience indirectly. This can help ground language in perceptual signals when interpreting diagrams, answering questions about images, or reasoning about events in video. section 4.5 section 4.5.1
Different signals are not simply mixed in their raw forms. Modality-specific encoders transform units such as text tokens, image patches, and audio segments into vector representations. These can be mapped into a shared space, where related signals from different modalities are positioned closer together. section 4.5.2
One way to learn that space is contrastive training on paired data: an image and its caption, for example, are brought closer together while unrelated pairs are pushed apart. The resulting alignment can support image retrieval from text, image-guided responses, or generation shaped by audio without first translating it into words. section 4.5.2
Figure 4.18 Learning a shared multimodal representational space through contrastive alignment. Text, images, and audio are mapped into a shared space. Related inputs across modalities are pulled closer together, while unrelated inputs are pushed apart, allowing cross-modal associations to be learned through relative distance.

The figure shows how related text, images, and audio are aligned in a shared space. section 4.5.2
Fusion determines when information from different modalities interacts:
- Late fusion combines representations near the output, after mostly separate processing. It is relatively simple and stable, but limits earlier cross-modal influence. section 4.5.3
- Early fusion merges representations soon after encoding and processes them jointly. It enables richer interaction but increases complexity and sensitivity to differences in scale, resolution, or noise. section 4.5.3
- Cross-attention in a shared Transformer backbone lets one modality selectively query another while retaining modality-specific processing paths. section 4.5.3
Figure 4.19 Early and late fusion strategies in multimodal models. In late fusion, each modality is processed mostly independently, and the representations are combined only near the output. In early fusion, modality-specific encoders produce representations that are merged earlier and then processed jointly by shared layers. The choice of fusion strategy determines how deeply modalities can influence one another during learning, trading simplicity and modularity for richer cross-modal interaction.

The figure contrasts early and late fusion and their effects on cross-modal interaction. section 4.5.3
Broader input does not by itself guarantee understanding. Captions, transcripts, and descriptions help organize perceptual signals in a linguistic frame, while training reinforces recurring associations. A model may associate smoke with fire without establishing that fire causes smoke; perceptual grounding alone is not causal explanation or evidence of learning through intervention. section 4.5.4 section 4.8












