Given a sequence, next-token prediction trains a model to predict each token from the tokens that precede it. This objective can be written as predicting token xₜ from context x₍₁:ₜ₋₁₎. The same continuation process can be placed in different task contexts by prompts. section 13.3.3 section 12.2
The objective does not specify each downstream task separately. Representations acquired during training can support summarization, translation, question answering, and programming; in-context examples can guide a model without changing its parameters. Greater scale expanded the range of patterns available for reuse. section 13.3.3
Attention helps construct context for prediction, but it is only one part of the system: model size, data, objectives, and decoding also shape its capabilities. Since training rewards plausible continuation rather than factual verification, fluent predictions can still be unsupported or incorrect. section 12.1.3 section 12.4.2
From context to continuation
A transformer walkthrough follows pronoun resolution from simplified embeddings through attention calculations to next-token prediction, then extends the process to factual association and chained generation. section 12.2












