Word tokens keep sequences short, but a vocabulary large enough to cover a huge dataset can become impractical. Restrict it too tightly and unfamiliar words collapse to [UNK]. Character tokens cover more inputs with a small vocabulary, but make sequences longer. Subword tokenization balances those costs by using a mixture of characters, word fragments, and sometimes whole words. section 14.2.1 section 14.2.2
Grow tokens from a tiny dataset
Take “the quick brown fox,” “the slow brown fox,” and “the quick brown foxhound.” First count words and split each into characters: brown becomes b r o w n. Count adjacent pairs, weighting each word by how often it occurs. The pair o w occurs in brown three times and slow once, so merge it into ow. Repeat with the most frequent available pair; later merges can join ow to other symbols. Common sequences become compact tokens without requiring every word to have its own entry. section 14.2.2
Listing 14.6 Running a few steps of byte-pair merging
def count_pairs(counts): pairs = collections.Counter() for word, freq in counts.items(): symbols = word.split() for pair in zip(symbols[:-1], symbols[1:]): pairs[pair] += freq return pairs def merge_pair(counts, first, second): split = re.compile(f"(?<!\S){first} {second}(?!\S)") #1 merged = f"{first}{second}" #2 return {split.sub(merged, word): count for word, count in counts.items()} for i in range(10): pairs = count_pairs(counts) first, second = max(pairs, key=pairs.get) counts = merge_pair(counts, first, second) print(list(counts.keys()))
The pair-counting and merging code shows how word frequencies select each new symbol. section 14.2.2
Think of the goal as useful compression: fewer tokens per example without an enormous vocabulary. After splitting, tokens are indexed as integers for a model. On one sample line, a 2,000-entry word tokenizer produces 13 tokens; a 2,000-entry subword tokenizer produces 16, while preserving the flexibility to represent words through smaller pieces. A character tokenizer produces 63 tokens for that line. section 14.2.1 section 14.2.2
Choose the tokenizer for the workload:
- For large pretrained sequence models, subwords avoid both very long character sequences and an unwieldy word vocabulary. RoBERTa uses a subword tokenizer with a vocabulary of about 50,000 terms. section 15.4.2
- For a small bag-of-words classifier, word tokens may be simpler and quicker to prepare; its weights also correspond to recognizable words. Subwords with a sufficiently large vocabulary remain a viable choice. section 14.4.1
