Misalignment is the gap between what a model is trained to optimize and what people later ask it to do. Predicting a plausible next token can produce fluent text without producing factual or useful information. section 2.10
A plausible but invented answer
When ChatGPT invented book titles, the response looked like a likely continuation of the prompt. That may have been a strong next-token prediction, even though the titles were false. Manual preference data can narrow this gap by encouraging helpful and safe behavior, but it does not remove it. section 2.10
The same problem appears in predictive modeling. A model trained to predict whether a product sells is not automatically a model of how changing its price affects sales. If historical prices barely vary, it may rely on color or star rating instead. Testing an unseen price can therefore generate a confident fiction with no corresponding training loss. section 2.11
Adding factual documents through retrieval can make responses more convincing, but it does not by itself change the model’s basic objective. The practical takeaway is simple: judge a model against the task you intend to use, not only the metric used during training. section 2.10 section 2.11
