A token is a unit an LLM uses to process a prompt or generate a response. Tokenization turns text into a sequence of tokens; depending on the tokenizer, a token may represent a character, word, subword, or larger unit. For English text, four characters or 0.75 words per token is only a rough estimate—not a reliable way to size an API request. section 2.8.2
In a conceptual LLM pipeline, prompt tokens become numerical embeddings. The model then generates a completion by predicting one token at a time, adding each new token to the sequence used for the next prediction. Generation can end at a special stop token or a configured maximum. Both prompt and generated tokens count toward a model’s context limit. section 2.8 section 11.4.1 section 6.7
Count before calling an API
With Python and OpenAI’s tiktoken, select an encoding and count the encoded units: `encoding = tiktoken.get_encoding("cl100k_base"); count = len(encoding.encode(prompt))`. The supplied example counts seven tokens for `I have a white dog named Champ` with that encoding. Use the encoding appropriate to the model rather than treating a word count as a token count. A completion API response separately reports `prompt_tokens` and `completion_tokens`: one example reports 32 and 26, for 58 total. section 2.8.3 section 3.2.1
Manage tokens as both a capacity limit and an operating cost:
- Leave room in the context window for the response. An overly long prompt reduces the available output length and may prevent generation; a prompt that is too short may omit needed context. section 6.7 section 2.8.3
- Keep prompts concise, set a generation limit, and use stop sequences where appropriate. Cache repeated prompts when that avoids another API call; consider a smaller model when it meets the task’s needs. section 3.4.5
- Track time to first token, time per output token, latency, and throughput in production. More tokens require processing and can increase latency and cost. section 11.4.1 section 2.8.2
