An autoregressive model generates a sequence one token at a time, predicting each next token from the prompt and all previously generated tokens. This causal chain maintains context and supports coherent, contextually appropriate output. section 5.1.1
The sequence probability is decomposed into conditional next-token probabilities, making the model's stepwise generation explicit. section 5.1.1
Autoregressive generation supports:
- Maintaining context across long passages while applying patterns learned during training. section 5.1.1
- Producing content, translations, and conversations through decoder models. section 5.7
- Using decoding algorithms such as greedy decoding, beam search, and sampling to balance coherence, diversity, and computational efficiency. section 5.7
Next-token completion
Given “The cat sat on the...”, the model evaluates possible next words and may select “mat” over “cloud” or “computer” because the preceding context makes it more appropriate. section 5.1.1







