DALL-E is a transformer-based generative model that converts natural-language descriptions into images. It treats text-to-image generation as sequence-to-sequence translation: words become a sequence of discrete image tokens, which are reconstructed into an image. section 12.0 section 12.1.1
The generation pipeline proceeds through three connected stages:
- Text encoding tokenizes the prompt and passes it through a pretrained BART encoder, producing contextual text embeddings. section 12.1.1
- Image-token prediction uses a transformer-based BART decoder to autoregressively predict visual tokens conditioned on the encoded prompt. section 12.1.1
- Image reconstruction sends the predicted tokens to a VQGAN decoder, which converts them into patches and assembles the final image. section 12.0
Training uses paired image–caption data. VQGAN encodes images as discrete tokens, while the transformer learns their correspondence with text through cross-entropy loss. At inference, the process reverses: a prompt produces image tokens, and the decoder reconstructs the result. section 12.1.1 section 1.2.2
Incremental image formation
A prompt such as “panda with top hat reading a book” demonstrates how the generated image develops as the model predicts 32, 64, and then 128 of its eventual 256 image tokens. section 12.0
Figure 12.1 Eight steps for building a text-to-image generator from scratch. This chapter focuses on the last step: replicating a transformer-based text-to-image generator such as DALL-E.
Use the eight-step implementation overview to situate transformer-based text-to-image generation within a complete from-scratch workflow. section 12.0
