An image-to-text transformer encodes an image and generates a natural-language caption. A vision encoder produces image embeddings, while a text decoder predicts caption tokens sequentially. section 4.3.2 section 4.4
Trace the captioning process through these stages:
- Represent the image with a vision encoder that processes image patches into embeddings. section 4.3.2 section 4.4.1
- Embed the input tokens and add positional encoding before passing them through decoder blocks. section 4.3.2
- Use masked self-attention to prevent future-token access and cross-attention to align text generation with image embeddings. section 4.3.2
- Predict the next vocabulary token, compare predictions with shifted caption targets, and minimize cross-entropy loss. section 4.3.2 section 4.4.1
Listing 4.7 The encoder–decoder transformer to add captions
class VisionEncoderDecoder(nn.Module):
def __init__(self, image_size, channels_in,
num_emb, patch_size=16,
hidden_size=128, num_layers=(6,6),
num_heads=4):
super(VisionEncoderDecoder, self).__init__()
self.encoder = VisionEncoder(
image_size=image_size, channels_in=channels_in,
patch_size=patch_size, hidden_size=hidden_size,
num_layers=num_layers[0], num_heads=num_heads)
self.decoder = Decoder(num_emb=num_emb,
hidden_size=hidden_size,
num_layers=num_layers[1], num_heads=num_heads)
def forward(self, input_image, target_seq, padding_mask):
bool_padding_mask = padding_mask == 0 #1
encoded_seq = self.encoder(image=input_image) #2
decoded_seq = self.decoder(input_seq=target_seq,
encoder_output=encoded_seq,
input_padding_mask=bool_padding_mask) #3
return decoded_seq
Use the encoder–decoder implementation to connect image encoding, caption decoding, vocabulary sizing, and configurable transformer depth. section 4.3.2
Training a captioning model
Train on Flickr 8k images, each paired with five human-written captions. With six encoder blocks, six decoder blocks, eight attention heads, 192-dimensional representations, and 8-pixel patches, the implementation has about 9.5 million parameters. Adam, cross-entropy loss, batches, and automatic mixed precision complete the training loop. section 4.2 section 4.4.1
