Before an LLM can use a text dataset, the text must become numbers. A tokenizer divides it into tokens that can be mapped to indices. The choice determines what information reaches the model: a convenient split on spaces may discard details that matter for code, names, or unfamiliar words. section 4.4 section 2.1.1
When splitting on spaces loses meaning
Consider code completion for `def add_two_numbers_together(x, y):`. A tokenizer that recognizes only whitespace-separated words may pass `[def, [UNK], y]` to the model. The function name and punctuation no longer provide useful clues. An unknown token is a fallback, not a harmless substitute for the original text. section 2.1.1
The main choices trade simplicity against the information retained:
- Word-based tokenizers commonly split on whitespace, though punctuation, dictionaries, and regular expressions are also options. They are easy to implement, but a manageable vocabulary needs a cutoff; unrecognized words then become unknown tokens. section 4.4.1
- Character-based tokenizers split text into characters and need a small dictionary. That simplicity is useful for an illustration, but character-based modeling is a weak choice for language modeling in the demonstrated training setup. section 4.4.1 section 5.2.1
- Subword-based tokenizers can split a word into smaller pieces: `jumped` becomes `jump` and `ed` in one example. Training a tokenizer for the intended data offers a way to reduce unknown tokens while retaining useful structure. section 4.4.1 section 4.4
Listing 4.10 Training your own subword tokenizers
import os
from pathlib import Path
import transformers
from tokenizers import ByteLevelBPETokenizer, SentencePieceBPETokenizer
from tokenizers.processors import BertProcessing
paths = [str(x) for x in Path("./data/").glob("**/*.txt")] #1
bpe_tokenizer = ByteLevelBPETokenizer() #2
bpe_tokenizer.train( #2
files=paths, #2
vocab_size=52_000, #2
min_frequency=2, #2
show_progress=True, #2
special_tokens=[ #2
"<s>", #2
"<pad>", #2
"</s>", #2
"<unk>", #2
"<mask>", #2
], #2
) #2
token_dir = "./chapters/chapter_4/tokenizers/bytelevelbpe/"
if not os.path.exists(token_dir):
os.makedirs(token_dir)
bpe_tokenizer.save_model(token_dir)
bpe_tokenizer = ByteLevelBPETokenizer(
f"{token_dir}vocab.json",
f"{token_dir}merges.txt",
)
example_text = "This sentence is getting encoded by a tokenizer."
print(bpe_tokenizer.encode(example_text).tokens)
# ['This', 'Ġsentence', 'Ġis', 'Ġgetting', 'Ġenc', \
# 'oded', 'Ġby', 'Ġa', 'Ġto', 'ken', 'izer', '.']
print(bpe_tokenizer.encode(example_text).ids)
# [2666, 5651, 342, 1875, 4650, 10010, 504, 265, \
# 285, 1507, 13035, 18]
bpe_tokenizer._tokenizer.post_processor = BertProcessing(
("</s>", bpe_tokenizer.token_to_id("</s>")),
("<s>", bpe_tokenizer.token_to_id("<s>")),
)
bpe_tokenizer.enable_truncation(max_length=512)
special_tokens = [
"<s>",
"<pad>",
"</s>",
"<unk>",
"<cls>",
"<sep>",
"<mask>",
]
sentencepiece_tokenizer = SentencePieceBPETokenizer() #3
sentencepiece_tokenizer.train( #3
files=paths, #3
vocab_size=4000, #3
min_frequency=2, #3
show_progress=True, #3
special_tokens=special_tokens, #3
) #3
token_dir = "./chapters/chapter_4/tokenizers/sentencepiece/"
if not os.path.exists(token_dir):
os.makedirs(token_dir)
sentencepiece_tokenizer.save_model(token_dir)
tokenizer = transformers.PreTrainedTokenizerFast(
tokenizer_object=sentencepiece_tokenizer,
model_max_length=512,
special_tokens=special_tokens,
) #4
tokenizer.bos_token = "<s>"
tokenizer.bos_token_id = sentencepiece_tokenizer.token_to_id("<s>")
tokenizer.pad_token = "<pad>"
tokenizer.pad_token_id = sentencepiece_tokenizer.token_to_id("<pad>")
tokenizer.eos_token = "</s>"
tokenizer.eos_token_id = sentencepiece_tokenizer.token_to_id("</s>")
tokenizer.unk_token = "<unk>"
tokenizer.unk_token_id = sentencepiece_tokenizer.token_to_id("<unk>")
tokenizer.cls_token = "<cls>"
tokenizer.cls_token_id = sentencepiece_tokenizer.token_to_id("<cls>")
tokenizer.sep_token = "<sep>"
tokenizer.sep_token_id = sentencepiece_tokenizer.token_to_id("<sep>")
tokenizer.mask_token = "<mask>"
tokenizer.mask_token_id = sentencepiece_tokenizer.token_to_id("<mask>")
tokenizer.save_pretrained(token_dir) #5
print(tokenizer.tokenize(example_text))
# ['_This', '_s', 'ent', 'ence', '_is', '_', 'g', 'et', 'tin', 'g', '_'
# 'en', 'co', 'd', 'ed', '_', 'b', 'y', '_a', '_', 't', 'ok', 'en',
# 'iz', 'er', '.']
print(tokenizer.encode(example_text))
# [814, 1640, 609, 203, 1810, 623, 70, \
# 351, 148, 371, 125, 146, 2402, 959, 632]
A Python example trains byte-level BPE and SentencePiece-style subword tokenizers from text files. Notice the explicit vocabulary size, minimum frequency, and special tokens: these are choices to make for the dataset, not incidental defaults. section 4.4.1
When building a model, choose the tokenizer alongside the data and embeddings rather than treating it as an interchangeable download. A Llama 3 implementation loads a matching tokenizer, adds `<PAD>` because its training batches need padding, and obtains the vocabulary for later model setup. It also defines encoding and decoding helpers, although the Hugging Face implementation already supports batch operations. If you experiment with another tokenizer, check what the model will see—not just whether tokenization runs faster. section 9.1.1 section 9.5
