DALL-E mini is an open-source project that approximates DALL-E’s text-to-image behavior with a transformer-based architecture and a VQGAN decoder. Its PyTorch reimplementation, min-DALL-E, exposes the process from prompt encoding to image reconstruction. section 12.1 section 12.5
Trace the generation pipeline through these stages:
- Encode each caption with a pretrained BART encoder to produce text embeddings. section 12.1.1
- Encode each training image with a pretrained VQGAN encoder into 256 discrete image tokens representing a 16 × 16 patch grid. section 12.1.1
- Train a BART decoder from scratch to autoregressively predict image tokens conditioned on the caption embeddings, minimizing cross-entropy loss. section 12.1.1
- At inference, tokenize the prompt, generate image tokens sequentially from a beginning-of-sequence token, and decode them with VQGAN into a color image. section 12.1.2
From prompt to pixels
For a prompt such as “a cat on a bicycle,” the BART encoder converts the text into embeddings. The decoder predicts one image token at a time until all 256 tokens are produced; VQGAN then reconstructs the corresponding image from its learned visual codebook. section 12.1.2 section 12.5
Figure 12.2 Training pipeline of the DALL-E mini model
Use the training-pipeline diagram to follow paired captions and images through BART and VQGAN into decoder training. section 12.1.1
Figure 12.3 The image generation process in min-DALL-E to generate an image from a text prompt
Use the inference diagram to connect prompt tokenization, autoregressive image-token sampling, and VQGAN decoding. section 12.1.2
