liveBook · Manning

Token

As explained in Timeless Algorithms, The Foundational Ideas

A next token is the linguistic unit that follows the current context in a sequence being modeled.

In a sequence, the next token is the linguistic unit still to be predicted. A neural language model uses the preceding context and statistical patterns learned during training as evidence, then a transformer assigns probabilities to possible continuations. This frames prediction as reducing uncertainty: the model narrows what might come next without removing every constraint. section 13.4.1

The prediction is constrained by:

  • Model capacity, training data, context, and computation, which limit the quality of the resulting prediction. section 13.4.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains how context and learned patterns yield a probability distribution, while naming constraints on prediction quality.
What you can do after
Understand how a neural language model uses context to assign probabilities to possible continuations.
How it connects
Connects neural language modeling to statistical inference, information theory, optimization, machine learning, and transformers.

Also explained in

Compare how 40 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.