A tokenizer converts text into tokens and then represents those tokens as numbers an LLM can process. Tokens may be whole words, word pieces, or subwords. Its vocabulary is the collection of available tokens, built by scanning training data and assigning each discovered token a unique identifier. section 2.2.1 section 2.6
A typical tokenization pipeline includes:
- Normalizing text into a standard representation, which can include changing case or standardizing Unicode byte encodings. section 2.6
- Segmenting the normalized text into words or subwords that can be processed independently. section 2.2.3
- Using byte pair encoding (BPE) to repeatedly find frequent character combinations, replace them with tokens, and add them to the vocabulary. section 2.2.3
- Balancing vocabulary size: more tokens can represent text efficiently, but increase computation, memory, and storage requirements. section 2.2.1
Why token counts behave unexpectedly
BPE may tokenize “running” as one token while splitting the slightly shorter “runnin” into “run” and “nin.” It favors frequent strings rather than counting characters, so sentence length does not reliably predict token count. The same implementation detail can affect tests, model evaluations, costs, and processing time. section 2.2.4
Figure 2.5 A simplified byte pair encoding algorithm for creating tokens: first, find the most frequent pair of characters “ng.” Next, replace all instances of “ng” with a placeholder token “T,” and add “ng” to the vocabulary. Repeat the process until no common byte pairs remain.
Use the BPE diagram to follow how frequent character pairs become vocabulary entries and how repeated merging produces word and subword tokens. section 2.2.3
Figure 2.8 A comparison of how two LLMs learn to perform arithmetic computations over time. Time is shown on the x-axis. The upper curve is a typical BPE tokenizer, while the lower curve is the same tokenizer modified to use tokens that represent individual digits. The y-axis describes the ability of the LLM to perform accurately, where a smaller number means fewer errors. The bottom line is that LLMs that use digit-level tokenization can learn how to do math better and faster.
Compare ordinary BPE with digit-level tokenization: representing each digit separately improves the learning and accuracy of arithmetic computations. section 2.3.2
Tokenization is not neutral. Homoglyphs can make visibly identical characters carry different encodings, creating confusion or security risks. Languages represented with finer-grained subwords may consume more tokens, increasing latency and usage charges. Programming code can still benefit from shared tokens for indentation, identifiers, operators, syntax, and comments. section 2.2.4 section 2.3.3 section 6.1
