Byte-pair encoding (BPE) builds subword tokens by repeatedly merging the most frequent adjacent token pair. It starts by splitting training text into characters and marking word endings, helping distinguish a sequence inside a word from a complete word. section 11.9
A simplified BPE pass follows this loop:
- Merge every occurrence of the most frequent pair into one token. section 11.9
- Repeat counting and merging until reaching a vocabulary size or a minimum frequency threshold. section 11.9
- Stop before rare merges create unintuitive tokens such as picturesorconversations. section 11.9
A small merge sequence
For the training paragraph, i followed by n is initially the most frequent pair, so occurrences become in. After recounting, e followed by r becomes the next merge; later merges produce tokens such as ic, th, ti, and ing. A resulting vocabulary contains both individual characters and learned subwords.
