Tokenization converts prompt or response text into a sequence of discrete tokens that an LLM can process. The model passes these tokens to embedding, then through its encoder-decoder generation pipeline. section 2.8 section 2.8.2
Treat tokens as an operational constraint when designing LLM applications:
- Control token volume because more tokens increase model processing, cost, latency, and resource requirements. section 2.8.2
- Expect token boundaries to vary by algorithm: tokens may represent characters, words, sub-words, or larger linguistic units. section 2.8.2
- Budget for both prompt and response tokens when estimating model limits and throughput. section 2.8.2
A sentence becomes eight tokens
Using an OpenAI tokenizer, “I have a white dog named Champ” is represented as eight tokens. During generation, the model predicts the next token, appends it to the sequence, and uses the expanded context to predict the following token. section 2.8.2
Tokenization also changes search behavior. In a Redis index, a URL stored as a TagField remains one exact-match tag; a TextField tokenizes and indexes its parts for word searches, so each field type must match the intended query. section 8.3
