Transformer-based text-to-image model

As explained in Build a Text-to-Image Generator (from Scratch)

A transformer-based text-to-image model generates images from text by modeling relationships between textual and visual token sequences with transformer architectures.

A transformer-based text-to-image model generates images by treating visual patches as a sequence and predicting them from text tokens. Unlike diffusion models, it generates patches sequentially from the top-left toward the bottom-right. section 1.7 section 1.2

The generation pipeline connects text and image representations through three main stages:

• Encode each training image with VQGAN as a sequence of discrete image tokens. section 1.2.2
• Process captions with the BART encoder and decoder to produce text tokens. section 1.2.2
• Train the BART decoder to predict text tokens matching the image tokens, then pass predicted tokens through the VQGAN decoder. section 1.2.2

Sequentially generating a prompted image

For “panda with a top hat reading a book,” the transformer predicts the first image patch from the prompt, then predicts each subsequent patch using the prompt and previously generated patches until all 256 patches are complete. section 1.2

Figure 1.3 How the min-DALL-E model generates an image based on the prompt “panda with a top hat reading a book.” The top-left subplot shows the output when 32 image patches are generated. The second subplot in the top row shows the output when 64 patches are generated. The remaining images show the outputs for 96, 128, . . ., and 256 patches.

Trace the panda image as its patch sequence grows through 32, 64, 96, 128, and up to 256 generated patches. section 1.2

Figure 1.5 How DALL-E mini is trained to generate an image based on a text prompt

Inspect the paired training and inference paths linking captions, BART, image tokens, and VQGAN reconstruction. section 1.2.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes

Explains tokenization, training alignment, inference, and sequential patches, but does not detail transformer internals.

Examples

Uses DALL-E mini and a panda wearing a top hat while reading a book as concrete generation examples.

What you can do after

Operate min-DALL-E to generate images from text prompts using its PyTorch implementation.

How it connects

Connects multimodal generative models with VQGAN, BART, diffusion models, latent diffusion models, and variational autoencoders.

liveBookmanning.com

Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Related concepts

• DALL-Eincludes · DALL-E is a family of OpenAI generative models that creates and edits images from natural-language descriptions and other conditioning inputs.
• transformer architectureuses · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• VQGANhas part · VQGAN is a generative model that combines vector-quantized discrete latent representations with adversarial and perceptual objectives to synthesize images.
• BARThas part · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder.
• text-to-image modelkind of · A text-to-image model is a generative machine-learning model that produces images conditioned on textual descriptions or prompts.

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.