A subword unit is a token fragment smaller than a full word. Byte pair encoding (BPE) uses subwords, and individual characters when necessary, to represent unfamiliar words without replacing them with an unknown token. section 2.5
In the GPT pipeline, BPE converts text into subword tokens, token IDs, and then embedding vectors that neural networks can process. This lets the tokenizer and model handle text containing words absent from training data. section 2.9
BPE builds and uses subword units by:
- Starting with individual characters in the vocabulary. section 2.5
- Iteratively merging frequent character combinations into subwords and frequent subwords into words. section 2.5
- Breaking unknown words into subwords or characters so they remain decodable. section 2.5
Unknown-word round trip
A BPE tokenizer can encode “someunknownPlace” or “Akwirw ier” as sequences of token IDs, including subwords and characters, then decode those IDs to reconstruct the original text. section 2.5
tokenizer = tiktoken.get_encoding("gpt2")
Inspect the GPT-2 tokenizer configuration used to produce subword token IDs. section 2.5
