A tokenizer converts natural-language text into smaller tokens and maps those tokens to integer identifiers that a language model can process. It also converts generated token identifiers back into response text. section 10.1.1
Token boundaries are not always whole words: punctuation, whitespace, and parts of larger words may become separate tokens. The resulting token count matters because a model can process only a limited context length. section 10.1.1 section 12.2.2
Use tokenizer awareness when preparing prompts or fine-tuning data:
- Choose a tokenizer that matches the model; otherwise, token identifiers may not align with the model’s parameters. section 12.2.2
- Check prompt and dataset lengths so entries do not exceed the context limit and become truncated or partially processed. section 12.2.2
- Run tokenization during preprocessing to translate training data from text into tokens before fine-tuning. section 12.2.4
List me five of the most populated cities in the world .
Preview the sentence used to demonstrate how words and punctuation are split into tokens, then compare its token identifiers in b6 and generated identifiers in b7. section 10.1.1
