Tokenization converts natural-language text into smaller, discrete tokens and then maps each token to an integer identifier that a language model can process. section 10.1.1
A prompt becomes token identifiers
The prompt “List me five of the most populated cities in the world.” can be split into tokens that preserve whitespace and punctuation. It then becomes a list of integers, with repeated words receiving the same identifier; the model processes those integers before generating a response. section 10.1.1
Tokenization matters when designing prompts or preparing fine-tuning data:
- Token boundaries are not fixed to whole words: tokenizers may split larger words or group smaller words. section 10.1.1
- The tokenizer must align with the model; mismatched token identifiers do not align with the model’s parameters. section 12.2.2
- Context length limits the number of tokens processed at once, so oversized prompts or training entries may be truncated and lose information. section 12.2.2 section 10.3
- Tokenization also converts generated model output back into response text. section 10.1.1
