liveBook · Manning

Tokenization

As explained in Understanding AI

Tokenization is the process of splitting text into reusable units called tokens for language-model processing.

Language models do not receive text as people read it. Tokenization splits text into reusable pieces called tokens: whole words, word parts, punctuation, numbers, or spaces. Each token then receives a numerical ID, a label the computer can process rather than a definition or meaning. section 2.1.1

Tokenization handles language that a fixed word dictionary would struggle with:

  • Short, common words may remain whole, while longer or less frequent words may split into smaller pieces. section 2.1.1
  • Reusable pieces help process new words, names, mixed languages, typos, and variations. section 2.1.1

A word can become several tokens

A person may read “unbelievable” as one meaningful word, while a model receives three tokens and three numerical IDs. The sentence “The performance was unbelievable.” may therefore contain a different token pattern than its visible words suggest. section 2.1.1

How Understanding AI covers it

How deep it goes
Explains token boundaries, numerical IDs, and vocabulary rationale, but does not detail the tokenization algorithm’s implementation.
Examples
Uses “unbelievable,” a full sentence, typos, other languages, code, and an online tokenizer.
What you can do after
Operate OpenAI’s online tokenizer to inspect token boundaries and token counts.
How it connects
Connects tokens with language models, numerical IDs, vocabulary, and text processing.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.