min-DALL-E is a compact Transformer-based text-to-image model that converts prompts into discrete image tokens and reconstructs them as images. It approximates DALL-E with a BART encoder, an autoregressive BART decoder, and a VQGAN decoder rather than predicting pixels directly. section 12.0 section 12.1.2
Trace generation through four connected stages:
- Tokenize the prompt and encode it with pretrained BART into dense text embeddings. section 12.1.1
- Start with a beginning-of-sequence token and have the trained BART decoder predict 256 image tokens autoregressively, one at a time. section 12.1.2 section 12.5
- Represent each token as a learned-codebook image patch instead of a raw continuous pixel array, reducing the transformer’s prediction space. section 12.1.2
- Pass the completed sequence to the pretrained VQGAN decoder, which reconstructs and stitches the patches into a color image. section 12.1.1
Prompt-to-image trace
For “panda with top hat reading a book,” the decoder progressively generates image tokens; visualizations after 32, 64, and 128 tokens show the image taking shape before all 256 tokens are decoded. section 12.0
Figure 12.3 The image generation process in min-DALL-E to generate an image from a text prompt
Follow the prompt, token-generation, and VQGAN reconstruction stages in the end-to-end inference diagram. section 12.1.2
