Next-token prediction generates a sequence one token at a time: the model predicts each next token from preceding tokens and any conditioning input. In a transformer, a causal mask prevents the decoder from using future tokens. section 2.2.2 section 4.1.2
Training and generation use the same next-token relationship in different ways:
- During training, shift the token sequence to create inputs and targets. Compare the decoder’s predictions with the target tokens using cross-entropy loss, then update model parameters. section 2.2.2 section 4.1.2
- During generation, begin with a start token, predict the next token, append it to the sequence, and repeat until an end token appears. section 2.4.2 section 4.1
Captioning an image
For the caption “a dog runs,” the input is “<start> a dog runs” and the target is “a dog runs <end>.” At inference, the decoder generates the caption one token at a time from image features and the tokens already produced. section 4.5 section 4.1.2
Figure 1.3 How the min-DALL-E model generates an image based on the prompt “panda with a top hat reading a book.” The top-left subplot shows the output when 32 image patches are generated. The second subplot in the top row shows the output when 64 patches are generated. The remaining images show the outputs for 96, 128, . . ., and 256 patches.
See how successive predictions build an image from patches, from the first 32 patches through the completed image. section 1.2
