Byte-pair encoding (BPE) tokenizes text as subwords rather than whole words. It builds a vocabulary by repeatedly merging frequent adjacent symbols, placing its units between individual letters and complete words. section 5.2.1
This compromise controls two competing costs: word-level tokenization creates a very large vocabulary, while alphabet-level tokenization keeps the vocabulary small but makes each sentence much longer in tokens. GPT uses BPE to encode text before processing tokens. section 5.2.1
The tokenization choices can be compared directly:
- Words: fewer tokens per sentence, but many distinct vocabulary entries. section 5.2.1
- Alphabets: a limited vocabulary of 26 symbols, but more tokens per sentence. section 5.2.1
- Subwords through BPE: a middle ground between those extremes. section 5.2.1













