A token count measures how many pieces of text an LLM receives as input or produces as output. These pieces may be words, fragments, or characters rather than whole words. section 2.2.4
Count does not simply follow character or sentence length. Byte-pair encoding greedily chooses familiar pieces, so a longer string can contain fewer tokens than a shorter variation. section 2.2.4
Token counts matter because they affect:
- Billing, when providers charge for input and output tokens. section 2.3.3
- Response time, which depends partly on how much text the tokenizer represents. section 2.3.3
- Language equity, because some languages require substantially more tokens than English for comparable requests. section 2.3.3
A counterintuitive count
GPT-3 tokenizes “I’m running” into one fewer token than “I’m runnin.” The complete word is common enough to have its own token, while the shortened form is split into pieces such as “run” and “nin.” section 2.2.4
Figure 2.6 Tokenizing two different sentences
Compare the tokenization of two nearly identical strings and see why token counts can reverse ordinary expectations about length. section 2.2.4
