liveBook · Manning

Next-word prediction

As explained in Timeless Algorithms, The Foundational Ideas

A language-modeling objective in which a model learns to predict the next token from preceding tokens in a sequence.

Given a sequence, next-token prediction trains a model to predict each token from the tokens that precede it. This objective can be written as predicting token xₜ from context x₍₁:ₜ₋₁₎. The same continuation process can be placed in different task contexts by prompts. section 13.3.3 section 12.2

The objective does not specify each downstream task separately. Representations acquired during training can support summarization, translation, question answering, and programming; in-context examples can guide a model without changing its parameters. Greater scale expanded the range of patterns available for reuse. section 13.3.3

Attention helps construct context for prediction, but it is only one part of the system: model size, data, objectives, and decoding also shape its capabilities. Since training rewards plausible continuation rather than factual verification, fluent predictions can still be unsupported or incorrect. section 12.1.3 section 12.4.2

From context to continuation

A transformer walkthrough follows pronoun resolution from simplified embeddings through attention calculations to next-token prediction, then extends the process to factual association and chained generation. section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Connects prediction to attention and autoregressive generation, but does not derive the loss mathematically.
Examples
Uses pronoun resolution, semantic association, factual recall, and chained generation to illustrate prediction.
What you can do after
Understand how a continuation objective can support varied tasks and why plausible predictions may still be unreliable.
How it connects
Links language modeling to attention, autoregressive generation, in-context learning, and transfer learning.

Also explained in

Compare how 13 books explain next-word prediction →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.