Byte-pair encoding (BPE) is a subword tokenization method adapted from data compression. It repeatedly merges the most frequent consecutive character pair, balancing vocabulary size against tokenized sequence length. section 11.2.1
Unlike character-level or word-level tokenization, BPE can represent common words or word fragments as single tokens while breaking uncommon words into smaller pieces. GPT-2 applies the resulting token indexes before word embedding. section 11.1.2 section 11.2.1
The practical BPE workflow is:
- Convert text into subword tokens. section 11.2.1
- Map each token to its corresponding integer index. section 11.2.1
- Repeat frequent-pair merges until reaching a desired vocabulary size or gaining no useful merges. section 11.2.1
Tokenizing and decoding a sentence
Using the BPE tokenizer on “This is the original text.” produces six tokens: ['This', ' is', ' the', ' original', ' text', '.'], with indexes [1212, 318, 262, 2656, 2420, 13]. Decoding those indexes reconstructs the original sentence. section 11.2.1
from utils.bpe import get_encoder example="This is the original text." #1 bpe_encoder=get_encoder() #2 response=bpe_encoder.encode_and_show_work(example) print(response["tokens"]) #3
Run the supplied Python example with get_encoder and encode_and_show_work to inspect both BPE tokens and their indexes. section 11.2.1
