An autoregressive model generates text one token at a time, predicting the next token from the preceding context. Repeating this prediction produces a complete sentence or passage that aims to be coherent and correct. section 5.1.2
This objective is powerful but directional: the model uses information from only one side of the predicted token. That restriction matters when a word depends on both preceding and following words. section 5.1.2
Completion versus conversation
A model trained on the raw language-modeling objective may continue text without recognizing the conversational intent behind a prompt. ChatGPT can respond more naturally because it was fine-tuned for human-like conversation, illustrating that next-token prediction and useful interaction are related but distinct goals. section 5.1.2
Figure 5.3 The objective of language modeling is to predict words based on their context.
Use the context-prediction illustration to see how preceding words guide the next-token objective. section 5.1.2
