liveBook · Manning

Token

As explained in Local Image Generation

A token is a text-processing unit used by an AI model that may represent a whole word or a smaller meaningful part of one.

A token is a text-processing unit used by an AI model. Unlike a character or a whole word, it can represent a smaller meaningful part of a word, helping the model turn prompts into numbers it can process. section 2.2.3 section 4.3.2

Keep these three ways of looking at text separate:

  • Characters are individual letters, symbols, or spaces. “sandworm” has eight characters, while “sand worm” has nine because of the space. section 2.2.3
  • Words are groups of characters usually separated by spaces; changing the space can change the meaning. section 2.2.3
  • Tokens sit between those ideas: they are closer to words, but may split words into smaller parts that capture useful meaning. section 2.2.3
  • Tokenization breaks a prompt into numbered pieces that can be passed into a neural network. section 4.3.2

A prompt-length surprise

In Stable Diffusion’s prompt interface, “racetrack” uses two tokens rather than one. The interface counts tokens, and the prompt limit shown here is 75 tokens. Adding a space, as in “race track,” can change how the text is split—and potentially how the prompt is understood. section 2.2.3

Figure 2.10 Token count for the word “racetrack”.

Inspect the token counter for “racetrack” to see that a familiar-looking word can occupy multiple tokens. section 2.2.3

Figure 4.17 Closer look at the CLIP encoding.

Trace where tokenization fits into the CLIP encoding path that transforms prompt text into numerical representations. section 4.3.2

How Local Image Generation covers it

How deep it goes
Defines characters, words, tokens, and tokenization, then stops before detailing the complete tokenizer or encoder implementation.
Examples
Uses “sandworm,” “sand worm,” “racetrack,” and “race track” to show character, word, spacing, and token differences.
What it assumes
Starts from basic programming notions of characters and strings, then introduces words, tokens, and tokenization.
What you can do after
Evaluate prompt token usage with a token counter and recognize why spacing or word structure changes tokenization.
How it connects
Connects tokens to tokenization, prompt limits, CLIP encoding, and neural networks.

Also explained in

Compare how 40 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Local Image Generation. The section links point to the parts of the book it draws on.