A model cannot work directly with raw text. Tokenization splits text into units—such as words, subwords, or characters—and maps them to numerical indices that the model can process. Encoding makes that conversion; decoding turns token indices back into text. The choice of units determines what information reaches the model during training and inference. section 2.1.1 section 4.4.1 section 9.1.1
When a useful name disappears
Suppose a code-completion model receives `def add_two_numbers_together(x, y):`. A tokenizer poorly matched to code may replace the function name with `<UNK>`, hiding a strong clue about what the function does. Frequent unknown tokens can indicate a vocabulary that is too small or a tokenizer unsuited to the task. section 2.1.1
Token boundaries also affect simpler models. Splitting a sentence on spaces leaves `words` and `words?` as separate entries in a bag-of-words count. That may be acceptable for a quick baseline, but it is a poor substitute for choosing a strategy that fits the data. Tokens need not match meaningful word parts: morphology can inform tokenization without dictating every boundary. section 2.2 section 2.1.1
Choose and check a tokenizer against the job it must do:
- Inspect how it splits representative inputs, especially punctuation, code, numbers, and domain-specific terms. Count or examine unknown tokens rather than assuming familiar-looking text survives encoding. section 2.1.1 section 2.5
- Decide whether to reuse a tokenizer, add important domain tokens, or train one on your own data. A general LLM tokenizer is a reasonable starting point; a specialized task may justify more control. section 4.4 section 9.1.1
- Check training requirements as well as inference behavior. Batch training may require a padding token absent from the tokenizer you loaded; keep the vocabulary and encoding and decoding behavior aligned with the model setup. section 9.1.1
Figure 4.3 Llama’s tokenization of the first arithmetic problem in the comparison table. Notice that each digit is highlighted individually, meaning that the model will eventually see all the digits.
The digit-by-digit view makes a consequential tokenization choice visible: arithmetic inputs need not be split the same way by different tokenizers. The accompanying model comparison also involves fine-tuning, so it does not isolate tokenization as the only cause of performance differences. section 4.4.1
Tokenization is one preparation step in a larger training pipeline: a dataset is preprocessed and converted to numerical input before the model learns from it. Treat the tokenizer as part of that pipeline, not as an interchangeable afterthought. A practical rule is to test its output on the inputs that matter most before committing to training. section 5.2.1 section 9.1.1
