An image-text pair links an image to an associated description, often called a caption. It is a training example, not a model component: the pairing tells a model which visual and textual content belong together. One image can form several pairs with different captions. Flickr 8k, for instance, contains about 8,000 images with five captions per image. section 4.1.1 section 8.2.1
Listing 8.1 Visualizing image–caption pairs in Flickr 8k
imgfolder=r"files/Images" import os with os.scandir(imgfolder) as fb: files=[f.name for f in fb] start=100 imgs=files[start:start+10] #1 dfi=df[df["image"].isin(imgs)].copy() dfi["length"]=dfi["caption"].str.len() dfi=dfi.sort_values(['image',"length"]) dfi=dfi.groupby("image").first() #2 import PIL from matplotlib import pyplot as plt plt.figure(dpi=200,figsize=(15,10)) for i in range(10): plt.subplot(5,2, i+1) #3 img=f"{imgfolder}/{dfi.index[i]}" nparray=PIL.Image.open(img) plt.imshow(nparray) plt.title(f"{dfi.iloc[i]['caption']}") #4 plt.axis("off") plt.tight_layout() plt.show()
The listing loads images and their caption rows, selects the shortest caption for each chosen image, and plots the resulting pairs. Selecting one caption is useful for display; the dataset retains five descriptions per image. section 8.2.1
The same pair supports different learning tasks:
- Image captioning treats the image as input and its caption as the target text. Resize the image, tokenize the caption, add <start> and <end>, and train a decoder to predict each next caption token from the image features and preceding tokens. A causal attention mask prevents it from reading future tokens. section 4.1 section 4.1.1
- CLIP encodes both sides into a shared embedding space. Within a batch, contrastive training increases similarity for matching image-text pairs and decreases it for nonmatching combinations. The resulting similarity can help select an image for a text prompt. section 8.1.1 section 8.1
- Text-to-image training reverses the prediction direction. In min-DALL-E, a VQGAN encoder turns the paired image into discrete tokens; an encoded caption conditions a BART decoder trained to predict those image tokens. section 12.1.1
Turn one pair into captioning targets
Pair an image of two bears fighting with “two bears fight.” After adding boundary tokens, use [<start>, two, bears, fight] as decoder input and [two, bears, fight, <end>] as its position-by-position target. The image supplies visual context; cross-entropy loss compares predicted tokens with the target. section 4.1.1
The pair’s role depends on the objective: it can supply a caption to generate, a positive match to recognize, or an image to reconstruct from text. Paired data also underlies the described latent-diffusion training setup, although its computational cost leads to using a pretrained model rather than training that model from scratch. section 8.0 section 9.1.2
