Byte-pair encoding (BPE) is a tokenizer technique that turns text into reusable word pieces. Rather than relying on hand-written rules, it learns common adjacent symbol sequences from data and adds them to a token vocabulary. section 2.2.3 section 2.6
A simplified BPE process works like this:
- Start with individual letters or bytes as tokens. section 2.2.3
- Count adjacent pairs and find a frequent combination, such as “n” followed by “g.” section 2.2.3
- Merge that pair into a new subword token, add it to the vocabulary, and repeat. section 2.2.3
- Stop when the vocabulary reaches a chosen size or no useful common pairs remain. section 2.2.3
From letters to “ing”
Suppose “n” and “g” frequently occur together. BPE merges them into “ng.” If “i” commonly precedes that new token, it can merge “i” and “ng” into “ing.” The tokenizer can then represent familiar word parts compactly instead of storing every complete word. section 2.2.3
Figure 2.5 A simplified byte pair encoding algorithm for creating tokens: first, find the most frequent pair of characters “ng.” Next, replace all instances of “ng” with a placeholder token “T,” and add “ng” to the vocabulary. Repeat the process until no common byte pairs remain.
Follow the simplified merge diagram from the frequent pair “ng” through replacement, vocabulary addition, and repetition. section 2.2.3
BPE is efficient but not intuitive. A longer string can produce fewer tokens than a shorter one when the longer form is frequent. For example, “I’m running” can be one token shorter than “I’m runnin,” because “running” may have a learned token while “runnin” is split into pieces such as “run” and “nin.” Different tokenizers can therefore produce different counts, creating practical software risks. section 2.2.4
