liveBook · Manning

Next-word prediction

As explained in Machines that Think

Next-token prediction is the task of predicting the token that follows a given sequence of preceding tokens.

Next-token prediction trains a language model to predict the token that follows a sequence of preceding tokens. Its performance is measured by how accurately it predicts that next token. This objective rewards alignment with patterns in training data, not direct verification of factual truth. section 6.1.2

During generation, a decoder-only model uses the current context to make some continuations more likely than others, then produces text token by token. Tasks such as summarization and translation can be framed this way, allowing one model structure to serve different purposes. section 4.1.8 section 6.4.1

The model’s learned parameters shape its predictions, but they are not a database from which facts are retrieved. At inference time, a fact can influence an answer only if it is encoded in those parameters or present in the active input. Earlier conversation turns, for example, must be reintroduced into the context to affect generation. section 6.4.1

Plausible continuation is not the same as factual correctness. The objective does not require a model to check external facts or express uncertainty, so weak or ambiguous patterns can produce confident hallucinations. Evaluation and fine-tuning practices that reward answers more than refusals can reinforce this tendency. section 6.1.2

Figure 6.1 Models learn patterns from training data and extend them to new cases. In this example, animals associated with water are often labeled as fish. When no clear example is available, as with the dolphin, the model does not stop or admit uncertainty. It continues by extending the strongest available pattern, incorrectly classifying the dolphin as a fish rather than a mammal.

An animal-classification example illustrates how extending a familiar pattern can produce a plausible but incorrect answer. section 6.1.2

From prediction to mathematics

A study using MathCAMPS tested models at different training stages. Models trained only through next-token prediction learned mathematical skills in an order correlated with a human-designed curriculum. The result suggests that prediction can support mathematical structure, without showing that models learn as children do. section 9.5.2

Fluent output alone does not establish human-like understanding or intention. At the same time, the ability to reuse learned structure across tasks leaves open a broader debate about what forms of intelligence such systems may exhibit. section 4.1.8 section 9.5.2

How Machines that Think covers it

How deep it goes
Explains the objective and its reliability implications without specifying a loss formula or implementation.
Examples
Illustrates pattern extension with animal classification and examines mathematical skill development using MathCAMPS.
What you can do after
Evaluate and understand why fluent continuations can be useful without guaranteeing factual correctness.
How it connects
Connects next-token prediction with hallucination, attention, transfer of learning, and mathematical reasoning.

Also explained in

Compare how 13 books explain next-word prediction →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.