Image-to-text generation creates natural-language descriptions from visual input. An encoder–decoder transformer converts an image into features, then generates a caption token by token, providing a more accessible foundation for text-to-image generation. section 4.0 section 4.5
Trace the captioning process in four stages:
- Split the image into patches and encode them with a vision transformer into feature representations. section 4.0 section 4.5
- Prepare each caption as input and target sequences by adding start and end tokens and shifting the target one position. section 4.5
- Feed image features and previously generated tokens into the transformer decoder to predict the next caption token. section 4.5
- Compare predictions with target tokens using cross-entropy loss, then update model parameters during training. section 4.5
Captioning a dog image
For an image whose caption is “a dog runs,” train with “<start> a dog runs” as input and “a dog runs <end>” as the target. A trained model may produce a different wording while still accurately describing the scene. section 4.5 section 1.1.2
Figure 4.1 Eight steps to build a text-to-image generator from scratch. In this chapter, you’ll reinforce and apply skills from steps 1 and 2: building a transformer that captions images. The encoder is a ViT, similar to what you created for image classification in chapter 3. The decoder generates text based on image features, much like the decoder you developed for German-to-English translation in chapter 2.
Use the architecture overview to connect the ViT image encoder with the transformer decoder and relate captioning to text-to-image generation. section 4.0
