liveBook · Manning

Tokenization

As explained in Applied Reinforcement Learning

Tokenization is the process of dividing text into discrete units called tokens.

Tokenization converts text into smaller units a language model can process. A token may be a word, word part, punctuation mark, space-like marker, or symbol; each token maps to an integer ID in the model’s vocabulary. section 10.2

A simple sentence

“Take aspirin today” might split into “Take”, “aspirin”, and “today”, though a real tokenizer may divide a word into smaller pieces. section 10.2

How Applied Reinforcement Learning covers it

How deep it goes
Defines token types, ID mapping, and an example split, but does not detail tokenizer internals.
Examples
Uses “Take aspirin today” to illustrate a possible split into tokens.
What you can do after
Understand how input text becomes token IDs before model processing.
How it connects
Connects text tokens to vocabulary IDs, embeddings, and next-token generation.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.