Tokenization divides text into the units a generative AI model processes. A token may be a whole word, part of a word, or punctuation, depending on the tokenization method. section 1.1
Treat tokens as the model’s limited working space: both prompts and responses consume the context window. A rough average is four tokens per three words, but limits vary widely across models and implementations. section 1.1
Watch for these practical effects:
- Long inputs can push earlier prompts, responses, or data out of the context window without an explicit warning. section 1.1
- Summarizing a conversation and its key findings can preserve important context when token limits are tight. section 1.1
