liveBook · Manning

Tokenization

As explained in Local Image Generation

Tokenization is the process of breaking text into smaller units that can be mapped to numeric identifiers for model processing.

Tokenization turns a prompt such as “a tall tree in the forest” into smaller parts that can be assigned numbers and processed by a neural network. Machines need this translation because machine learning ultimately operates on numbers. section 4.3.2

Two straightforward ways to split text illustrate the trade-offs:

  • Character tokenization assigns numbers to each character, including spaces. It is simple and can work surprisingly well, but individual characters carry little meaning. section 4.3.2
  • Word tokenization assigns numbers from a large vocabulary. Words preserve more meaning, but representing every possible English word requires tens or hundreds of thousands of numbers. section 4.3.2

A tiny prompt experiment

Take “All mimsy were the borogoves” from Jabberwocky. A tokenizer might represent it character by character or map its words through a vocabulary; the resulting number sequence feeds the model. section 4.3.2

How Local Image Generation covers it

How deep it goes
Compares character and word representations, including their meaning and vocabulary trade-offs.
Examples
Uses a Jabberwocky sentence, ASCII character values, word vocabulary mapping, and learned text splitting.
What you can do after
Evaluate how character-based and word-based splits affect meaning and vocabulary size before model processing.
How it connects
Connects text prompts with the CLIP encoder, Stable Diffusion, neural networks, ASCII, and vocabulary lists.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Local Image Generation. The section links point to the parts of the book it draws on.