A multimodal model processes or connects two or more data modalities, including text, images, audio, and video. Unlike a unimodal model, it can transform one modality into another, such as generating an image from text or captions from an image. section 1.1 section 1.1.1
Common multimodal generation paths follow different input, transformation, and output patterns:
- Text-to-image models interpret a text prompt and produce a corresponding image. section 1.1
- Image-editing systems combine text and an existing image to produce a modified image. section 1.1.1
- Image-captioning transformers encode an image, combine its representation with caption tokens, and decode tokens until an end marker. section 4.1
- Transformer-based image generators divide images into patches and predict those patches as a sequence conditioned on text. section 1.7
- Diffusion-based generators begin with noise and iteratively denoise it according to a text prompt; latent diffusion performs this process in a lower-dimensional space before decoding the result. section 1.7
Captioning an image
After training on image-caption pairs, an image-to-text transformer receives an image and begins generation with <start>. It predicts one token at a time, appends each prediction to the input sequence, and stops at <end>. section 4.1
Figure 1.1 Comparison of unimodal and multimodal models
Compare unimodal pipelines, where input and output share one data type, with multimodal pipelines that connect different modalities. section 1.1.1
