Ask an AI system to describe a photo, then continue the conversation by voice: the interaction crosses several kinds of information. A multimodal system can process or generate content across multiple *modalities*, such as text, images, audio, and video. Unlike a text-only large language model (LLM), it may accept one kind of input and produce another. That does not mean every multimodal model handles every modality. section 2.4 section 1.16
Connecting modalities requires more than placing their data side by side. In one common approach, separate encoders turn images and text into representations that can be aligned. Training on paired images and captions can bring related representations closer together while separating unrelated pairs. This helps a model connect visual and textual concepts; other models use different approaches. section 2.4
Figure 2.4 During contrastive learning, similar image–text pairs are encoded close together in a shared embedding space, while dissimilar pairs are mapped farther apart.
See how paired image and text representations are positioned during contrastive learning. section 2.4
From a picture to a description
A text-only model might produce a description of a dog from a written prompt. A multimodal model can instead examine a dog’s image and describe what it sees, or generate an image from a text prompt. GPT-4 accepted text and image inputs; GPT-4o later enabled conversations mixing voice, images, and text. section 2.4 section 1.16
Multimodal interaction can make AI interfaces more flexible, but its outputs still warrant scrutiny. A text-to-video system, for instance, can produce realistic-looking clips while showing spatial inconsistencies or violations of physics. As interfaces increasingly mediate access to information, their errors—and the choices they make about what to show—also raise questions about transparency and influence. section 1.16 section 10.2
