A word tokenizer can need a large vocabulary to recognize names, product terms, and unusual spellings. Byte pair encoding (BPE) takes a different route: it builds reusable subword tokens from frequent adjacent symbols. This can keep a vocabulary manageable while still representing words the tokenizer has not seen as whole words. section 2.2.1 section 9.1.1
The basic training idea is:
- Start with small symbols, such as characters or bytes, and count which adjacent pairs occur often in a training corpus. section 2.2.1
- Merge a frequent pair into a token, then repeat to build a vocabulary of useful pieces. Fitting that vocabulary requires multiple passes over the text. section 2.2.1
- Use the learned pieces to encode new text. Unlike a whole-word vocabulary, the pieces can be combined to represent unfamiliar strings. section 2.2.1
A character-based BPE tokenizer starts from characters; byte-level BPE starts from 8-bit bytes. The latter has a base vocabulary of 256 tokens and can compose Unicode text from those bytes, although Unicode punctuation needs special handling. BPE is one subword approach alongside WordPiece and SentencePiece. Word tokens remain easier for people to inspect; subword tokens can be more useful when flexible prediction or generation matters. section 2.2.1 section 2.2
>>> import pandas as pd >>> from sklearn.feature_extraction.text import CountVectorizer >>> vectorizer = CountVectorizer(ngram_range=(1, 2), analyzer='char') >>> vectorizer.fit(texts) CountVectorizer(analyzer='char', ngram_range=(1, 2))
CountVectorizer configured for one- and two-character sequences provides a small experiment in the statistics behind subword training. It counts candidate pieces; it is not, by itself, a trained BPE tokenizer. section 2.2.1
Inspect candidate pieces
In the character-count experiment, the pair “uc” occurs 20 times and “an” occurs 19 times across the sample texts. Counts like these help identify recurring pieces. They do not alone show the final BPE vocabulary: merging changes which adjacent symbols are available to count next. section 2.2.1
The vocabulary also shapes what a language model predicts. GPT-2 uses byte-level BPE and has 50,257 tokens; its output scores correspond to tokens, not necessarily complete words. A decoded next-token candidate can be “ new” or “ non”, with a leading space included in the token. This makes completing a prompt that ends partway through a word possible. section 2.2.1 section 10.2.1
Listing 10.6 Finding the token with maximum probability
>>> encoded_prompt = tokenizer('NLP is a', return_tensors="pt") #1
>>> encoded_prompt = encoded_prompt["input_ids"]
>>> encoded_prompt = encoded_prompt.to(DEVICE)
>>> output = vanilla_gpt2(input_ids=encoded_prompt)
>>> next_token_logits = output.logits[0, -1, :]
>>> next_token_probs = torch.softmax(next_token_logits, dim=-1)
>>> sorted_ids = torch.argsort(next_token_probs, dim=-1, descending=True)
>>> tokenizer.decode(sorted_ids[0]) #2
' new'
>>> tokenizer.decode(sorted_ids[1]) #3
' non'
The GPT-2 inspection decodes two likely next-token candidates, making their leading spaces—and the distinction between tokens and whole words—visible. section 10.2.1
The practical choice is a trade-off: train BPE on the text your application must handle when vocabulary size and coverage matter, but use word tokens when human-readable features are the priority. section 2.2.1 section 2.2
