A multimodal model accepts more than one data modality, such as text, images, or audio. Multimodal Transformers extend sequence modeling beyond text-only inputs and support tasks such as image understanding and text-to-image generation. section 1.3.3 section 1.3
Text-to-image models show the idea directly: a text prompt conditions a neural network that progressively transforms noise into an image. Diffusion models provide the related process of adding noise during training and learning to remove it during generation. section 1.3.3
A multimodal system commonly combines these steps:
- Encode different inputs, such as an image and its textual description, into representations that can be compared. section 15.5.1
- Align matching image–text pairs while separating nonmatching pairs through contrastive learning. section 15.5.1
- Use text embeddings or other conditioning information to guide iterative generation toward a corresponding image. section 15.5.1
From prompt to image
For “an astronaut in a space suit riding a unicorn,” a text encoder produces an embedding. A conditioning vector guides a U-Net denoiser as it repeatedly removes noise until a clean image emerges.
