A text-to-image model generates an image that corresponds to a written description. It is a multimodal generative model: text is the input modality and an image is the output. That distinguishes it from a unimodal model that stays within one modality, such as a text model that produces more text. Some text-to-image models can also take an existing image alongside a prompt and edit it. section 1.1 section 1.1.1
Two approaches turn the prompt into a visual result:
- A transformer-based generator treats the image as a sequence. A text encoder represents the prompt; an autoregressive decoder predicts image tokens one after another; and an image decoder reconstructs the result. In min-DALL-E, these roles are filled by a BART encoder, a trained BART decoder, and a VQGAN decoder. Training uses paired captions and images. section 12.1.1
- A diffusion-based generator starts with random noise and repeatedly denoises it under guidance from the prompt. Forward diffusion adds noise to training images; the learned reverse process removes it. A latent diffusion model performs these steps in a compact representation rather than directly on a high-resolution image, then uses a variational autoencoder to produce the final image. section 1.3 section 1.7
In a latent diffusion pipeline, image–caption pairs provide the training signal. A variational autoencoder learns a compact image representation, a text encoder turns the caption into an embedding, and a denoising U-Net learns to predict noise conditional on the text. At generation time, the pipeline encodes the prompt, denoises random latent noise, and decodes the resulting latent image. Training a full latent diffusion model is computationally expensive, so a pretrained model can be used to generate high-resolution results. section 9.1.2 section 1.4
Figure 9.3 How a trained LDM generates an image based on a text prompt. The text prompt shown at the top-left corner is first encoded into a text embedding. To generate an image in the lower-dimensional space (latent space), we start with an image of pure noise (bottom left). We use the trained U-Net to iteratively denoise the image, conditional on the text embedding so the generated image matches the text prompt, with the guidance of a trained CLIP model. The generated latent image (bottom right) is presented to a trained VAE to convert it to a high-resolution image, which is the final output (top right).
Follow the prompt embedding into the denoising U-Net, then the generated latent image through the variational autoencoder to the final image. section 9.1.2
Watch an image take shape
For the prompt “panda with a top hat reading a book,” a min-DALL-E implementation can display intermediate results after 32, 64, or 128 of its eventual 256 image tokens. The partial outputs make sequential generation visible rather than treating the final image as a black box. section 12.0
Text-to-image models support visual prototyping, illustration, educational diagrams, synthetic training images, and text-guided edits. Their results still need scrutiny: models can produce inconsistent geometry, reflect social bias, or generate misleading images. Training-data copyright disputes and the computational cost of large models also matter when choosing how to build or use one. section 1.1.2 section 1.5 section 1.5.2 section 1.6
