Image tokenization turns visual content into a sequence of discrete integers that a transformer can predict. It differs from simply treating image patches as tokens: a vision transformer can embed patches for classification, while a VQGAN assigns compact visual representations to entries in a finite codebook so an image can later be reconstructed from their indices. section 3.5 section 11.0
The VQGAN path runs from pixels to tokens and back:
- Encode the image into continuous latent representations of its patches. section 11.0
- Quantize those representations using a finite codebook. Each selected entry becomes an integer token; one example sequence begins (873, 862, 873, …, 148). section 1.2.1
- Pass the code indices through the VQGAN decoder to reconstruct an image. Quantization approximates continuous representations rather than preserving every pixel value. section 1.2.1
This representation makes text-to-image generation a sequence-prediction task instead of requiring a transformer to predict raw pixels. In min-DALL-E, paired images supply VQGAN image tokens during training, while a BART encoder processes their captions and a BART decoder learns to predict the corresponding tokens. At generation time, the decoder predicts 256 tokens one at a time for a 16 × 16 patch grid; the VQGAN decoder turns the completed sequence into an image. section 12.1.1 section 12.1.2
Trace a generated image
For “panda with a top hat reading a book,” inspect the output after 32, 64, and 128 tokens, then after all 256. The partial outputs make the sequential prediction process visible. To try another prompt, use “an underwater fantasy city” when preparing the image-token tensor and attention state. section 12.0 section 12.3.2
In the min-DALL-E implementation, initialize storage for the 256 predicted tokens, an attention mask, and an attention-state cache. At each step, sample the next token using the encoded prompt and previous token, then retain the updated state. Temperature, top-K sampling, and a conditioning factor affect the resulting sequence. Here, pretrained DALL-E mini weights are converted for a PyTorch implementation; that is different from training the image tokenizer and generator anew. section 12.3.2 section 12.1.2
Listing 12.7 Predicting image tokens iteratively
from copy import deepcopy
imgtokens=[] #1
for i in range(256): #2
torch.cuda.empty_cache()
with torch.amp.autocast("cuda",dtype=getattr(torch, dtype)):
image_tokens[:, i + 1], \ #3
attention_state = decoder.sample_tokens(
settings=settings,
attention_mask=attention_mask,
encoder_state=encoder_state,
attention_state=attention_state,
prev_tokens=image_tokens[:, [i]],
token_index=token_indices[[i]])
imgtokens.append(deepcopy(image_tokens))
The sampling loop records each successive token sequence, making intermediate reconstructions possible. section 12.3.2
