When generating text, a model must decide what comes next without seeing future words. An autoregressive model predicts each next token from the tokens already available, making future predictions depend on past data. section 2.3
Training usually uses self-supervision: the model repeatedly guesses the next token in large text collections and adjusts its weights using the correct token. This differs from bidirectional prediction, which can use context on both sides of a missing word and is aimed more at understanding context than generating sequences. section 2.3 section 2.1
Autoregressive modeling appears across the language-model pipeline:
- Pretraining establishes foundational knowledge through next-token prediction over a very large corpus. section 2.1
- Post-training can refine the model for conversation, instruction following, or code generation. section 2.1
From prompt to continuation
Given “A good location for a romantic honeymoon is …,” the model predicts a likely continuation at the end of the prompt. GPT-family models and Google’s PaLM are examples of autoregressive models. section 2.3 section 2.17
Figure 2.1 The high-level training process for LLMs
Use the high-level training-process figure to place autoregressive pretraining alongside later post-training stages. section 2.1
