A token is a text-processing unit used by an AI model. Unlike a character or a whole word, it can represent a smaller meaningful part of a word, helping the model turn prompts into numbers it can process. section 2.2.3 section 4.3.2
Keep these three ways of looking at text separate:
- Characters are individual letters, symbols, or spaces. “sandworm” has eight characters, while “sand worm” has nine because of the space. section 2.2.3
- Words are groups of characters usually separated by spaces; changing the space can change the meaning. section 2.2.3
- Tokens sit between those ideas: they are closer to words, but may split words into smaller parts that capture useful meaning. section 2.2.3
- Tokenization breaks a prompt into numbered pieces that can be passed into a neural network. section 4.3.2
A prompt-length surprise
In Stable Diffusion’s prompt interface, “racetrack” uses two tokens rather than one. The interface counts tokens, and the prompt limit shown here is 75 tokens. Adding a space, as in “race track,” can change how the text is split—and potentially how the prompt is understood. section 2.2.3
Figure 2.10 Token count for the word “racetrack”.

Inspect the token counter for “racetrack” to see that a familiar-looking word can occupy multiple tokens. section 2.2.3
Figure 4.17 Closer look at the CLIP encoding.

Trace where tokenization fits into the CLIP encoding path that transforms prompt text into numerical representations. section 4.3.2







































