Byte pair encoding (BPE) tokenizes text into subword units by repeatedly merging frequent symbol pairs to build a vocabulary. It sits between raw text and the token IDs that feed GPT-style transformer models, reducing the need for unknown-word tokens. section 2.0 section 2.9
The tokenization flow has three practical properties:
- Represents familiar words with vocabulary entries while decomposing unfamiliar words into smaller subwords or individual characters. section 2.5
- Avoids relying on an <|unk|> token, allowing the tokenizer to represent any encountered word. section 2.5 section 2.4
- Produces integer token IDs that are later converted into embeddings for neural-network processing. section 2.9
Using the GPT-2 tokenizer
With tiktoken, select the GPT-2 encoding, call encode on text containing <|endoftext|> and an unfamiliar word, then call decode to reconstruct the text. The GPT-2 vocabulary contains 50,257 entries, with <|endoftext|> assigned ID 50256. section 2.5
Figure 2.11 BPE tokenizers break down unknown words into subwords and individual characters. This way, a BPE tokenizer can parse any word and doesn’t need to replace unknown words with special tokens, such as <|unk|>.
Shows how BPE decomposes unknown words into subwords and individual characters instead of replacing them with <|unk|>. section 2.5
