An autoregressive model generates an ordered sequence one token at a time. Each prediction depends on the tokens already generated and, in an encoder–decoder transformer, information supplied by the encoder. The decoder adds its prediction to the sequence, then uses the longer sequence for the next step. Text generation stops when it predicts an end token. section 2.1.2 section 4.1.2
The same loop can produce different kinds of output:
- For German-to-English translation, tokenize and encode the German sentence. Start the English decoder with BOS, predict its next token, append that token, and repeat until EOS. A subsequent mask is passed to the decoder during generation. section 2.4.2
- For image captioning, encode the image into embeddings. The decoder uses those embeddings and the caption so far, starting with <start> and stopping at <end>. An image of a skier can yield “a woman skis” through successive token predictions. section 4.1.2
- For text-to-image generation, encode the prompt and predict a sequence of discrete image tokens. A VQGAN decoder then turns those tokens into image patches and a complete image; the image tokens are not themselves pixels. section 12.1.1 section 12.3.2
To implement the image-token loop, prepare the encoded text and decoder state, then iterate over 256 token positions. At each position, pass the previous token and current state to the decoder, store the predicted token, update the state, and save a copy if you want to inspect the image as it forms. Sampling settings include temperature, top-k, and a factor controlling adherence to the prompt. section 12.3.2
Listing 12.7 Predicting image tokens iteratively
from copy import deepcopy
imgtokens=[] #1
for i in range(256): #2
torch.cuda.empty_cache()
with torch.amp.autocast("cuda",dtype=getattr(torch, dtype)):
image_tokens[:, i + 1], \ #3
attention_state = decoder.sample_tokens(
settings=settings,
attention_mask=attention_mask,
encoder_state=encoder_state,
attention_state=attention_state,
prev_tokens=image_tokens[:, [i]],
token_index=token_indices[[i]])
imgtokens.append(deepcopy(image_tokens))
Follow the 256-iteration decoder loop: each call supplies the previous image token and attention state, receives an updated state, and saves the intermediate token sequence. section 12.3.2
Saved token sequences can be decoded into a grid or animation. After 32 tokens, only the top eighth of the image is present; at 128, half is recognizable; at 256, the result is complete. This makes the progression inspectable, unlike a GAN that generates an image in one pass. section 12.4.2
