liveBook · Manning

Token

As explained in Applied Reinforcement Learning

A token is a discrete unit of text, such as a word, word fragment, punctuation mark, or symbol, processed by a language model.

A token is a discrete unit of text processed by a language model. It can be a whole word, part of a word, punctuation, a space-like marker, or a symbol. section 10.1 section 10.2

A possible token split

“Take aspirin today” might become three tokens: “Take”, “aspirin”, and “today”. Real tokenizers may split some words into smaller pieces, and each token is mapped to an integer ID. section 10.2

When generating language, the model chooses one token at a time from its vocabulary. The prompt and all previously generated tokens form the current state; adding a token extends that sequence. section 10.1 section 10.3

Figure 10.1 Overview of large language models architecture.

The architecture diagram situates tokenization between raw text and the model’s next-token prediction. section 10.2

How Applied Reinforcement Learning covers it

How deep it goes
Defines token forms, explains vocabulary IDs, and connects token selection to sequential language generation.
Examples
Uses “Take aspirin today” to illustrate a possible token split and describes generating a response one token at a time.
What it assumes
Assumes familiarity with neural networks and sequence models while defining tokenization and token units directly.
What you can do after
Understand how a language model converts text into token IDs and selects each next-token action.
How it connects
Connects tokenization, vocabulary IDs, next-token prediction, and Markov decision processes.

Also explained in

Compare how 40 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.