Byte-pair encoding (BPE) converts text into subword units by repeatedly merging frequent adjacent symbol pairs. Training a tokenizer on task-specific data can reduce unknown tokens and preserve more encoded meaning than relying on a generic tokenizer. section 4.4
BPE occupies a practical middle ground between character- and word-based tokenization: it keeps dictionaries smaller than word tokenizers while retaining more semantics than character tokenizers. The best split strategy depends on the task, so tokenizer choices can create downstream consequences. section 4.4.1
Tune the tokenizer around the data and the model:
- Set vocab_size to balance multilingual or unusual-token coverage against computational complexity. section 4.4.1
- Set min_frequency to control which subwords enter the vocabulary; higher values filter rare tokens but can discard useful ones. section 4.4.1
- Define special_tokens such as padding, unknown, mask, start, and end markers for model training. section 4.4.1
- Choose BPE or SentencePiece according to the base or fine-tuned model and the project’s requirements. section 4.4.1
Training a custom BPE tokenizer
A minimal Python workflow gathers text files and trains ByteLevelBPETokenizer with a 52,000-token vocabulary, minimum frequency 2, and explicit special tokens. section 4.4.1
Table 4.3 Tokenization allows GOAT 7B to outperform GPT-4 in math
|
Prompt
|
GOAT 7B
|
GPT-4 1.7T
|
Correct
|
|---|---|---|---|
|
3978640188 + 42886272 =
|
4021526460
|
4,021,526,460
|
4,021,526,460
|
|
4523646 minus 67453156
|
–62929510
|
–63,930,510
|
–62,929,510
|
|
Calculate 397 × 4429
|
1758313
|
1,757,413
|
1,758,313
|
|
What is 8914/64?
|
139 R 18
|
139.15625
|
139.28125 Or 139 R 18
|
Use the arithmetic comparison to inspect how tokenization affects numerical reasoning and how different token boundaries can produce different answers. section 4.4.1
