Sequence prediction estimates or generates ordered elements one at a time. Unlike single-output prediction, it produces a sequence whose later steps depend on earlier context, such as tokens in translation or image patches in generation. section 2.1.2 section 1.7
A transformer sequence-prediction pipeline typically proceeds as follows:
- Encode the input sequence into representations that capture its meaning or cross-element relationships. section 2.1.2 section 2.1
- Condition the decoder on the encoded input and the tokens already generated. section 2.1.2
- Predict the next element, append it to the input sequence, and repeat until an end-of-sequence token appears. section 2.1.2
German-to-English translation
Given “Wie geht es dir,” the decoder begins with BOS, predicts “How,” then uses BOS How to predict the next token. It stops when it predicts EOS. section 2.1.2
Text-to-image generation
A text-to-image model can treat an image as a sequence of patches, predicting patches from top-left to bottom-right. A DALL-E-style system instead encodes text, predicts image tokens autoregressively, and reconstructs them into image patches. section 1.7 section 12.1.1
