What happens when you show an AI a screenshot and ask why the code fails? A multimodal model can work with more than one kind of information in one system, connecting language with images, audio, video, documents, or code. This makes interactions less limited to typing text alone. section 1.3.2 section 1.5
For example, you can upload an image and ask questions about it, provide a screenshot for debugging help, generate an image from written instructions, or combine voice and text in one conversation. The important shift is that the model can relate information across formats rather than treating each format as a separate task. section 1.3.2
Underneath these varied interactions, the central pattern remains familiar:
- A transformer architecture connects relationships across different kinds of data. section 1.3.2
- Inputs are no longer restricted to text; models can be trained on language, images, audio, video, and documents. section 2.2.4
- Tokens, embeddings, attention, and prediction remain core ideas even when a model analyzes an image or answers about a performance. section 2.2.4
- The system still predicts what comes next one token at a time, rather than thinking or understanding like a human. section 2.2.4 section 1.5
A screenshot question
A person can provide a screenshot and ask for debugging help. The visual information and the written question appear together in the conversation, allowing the system to use both forms of input when responding. section 1.3.2
Multimodal output has improved as models and training have grown. Early image-generating systems could produce blurry images, distorted details, or nonsensical text inside visuals. More recent systems can produce more coherent, realistic scenes with accurate details, natural lighting, and coherent composition. Strong-looking output is still pattern-based generation, not proof of human-like understanding. section 1.3.2 section 1.5
Figure 1.4 Early multimodal AI outputs often contained noticeable errors, such as misspelled labels like VIDUAL instead of VISUAL, highlighting the limitations of early models in generating accurate text within images, even when the overall structure appeared coherent.

Compare an early generated visual whose overall structure is coherent but whose embedded label is misspelled, showing that plausible output can still contain important errors. section 1.3.2
Figure 1.5 Modern image generation models can produce highly realistic scenes, with accurate details, natural lighting, and coherent composition, often making the generated images appear almost indistinguishable from real photographs.

Inspect a more recent image-generation result for the greater realism, detail, lighting, and composition associated with improved multimodal output. section 1.3.2












