Byte-pair encoding (BPE) is a subword tokenization technique that represents text with reusable pieces rather than requiring every complete word to appear in the vocabulary. It supports broad vocabularies while handling unfamiliar words, languages, and word forms more flexibly. section 5.2.1 section 7.1
BPE is useful because it:
- Reduces the out-of-vocabulary problem by splitting unfamiliar words into recognizable subunits. section 5.2.1
- Preserves more internal word information than word-level tokenization, helping with rare words, misspellings, morphology, and compound words. section 5.2.1
- Provides the token representation used by transformer-based models such as BERT and GPT, including GPT-3.5. section 5.2.1 section 7.1
Tokenizing words into subwords
The word “exhilarating” is shown as four subword tokens: “ex”, “h”, “ilar”, and “ating”. The word “unhappiness” is shown as three tokens, illustrating how one word can be represented through smaller reusable pieces. section 5.2.1 section 7.1
Figure 5.8 Example of subword-level tokenization. The single word "exhilarating" is tokenized into four tokens.
View the subword-tokenization figure for the “exhilarating” example. section 5.2.1
