A vocabulary is the finite set of tokens a tokenizer can assign integer indices. Build it from the text you care about, then use it to turn text into numbers a model can process. A token absent from the vocabulary can be assigned an unknown-token index instead of stopping the pipeline. section 14.2
Index a short sentence
In R, set `vocabulary <- c("the", "quick", "brown", "fox", "jumped", "over", "dog", ".")`, split “The quick brown fox jumped over the lazy dog.” into words, then call `match(words, vocabulary, nomatch = 0L)`. The result begins `0, 2, 3, 4`: “The” does not match lowercase “the,” while “quick” has index 2. “lazy” also receives 0. Lowercasing before lookup would avoid the first mismatch. section 14.2
Vocabulary size depends on what counts as a token:
- Characters keep the vocabulary small but produce long sequences. In a full-text Moby Dick example, 89 character terms cover the text. section 14.2.1
- Words produce shorter sequences but require many terms. A 2,000-term word vocabulary does not cover every word in the same text; full coverage would require about 17,000 terms. section 14.2.1
- Subwords aim for shorter sequences without an enormous vocabulary. Byte-pair encoding starts with characters and adds tokens by repeatedly merging common adjacent pairs. section 14.2.2
Listing 14.5 Computing a byte-pair encoding vocabulary
compute_sub_word_vocabulary <- function(dataset, vocab_size) {
dataset <- split_chars(dataset)
vocab <- compute_char_vocabulary(dataset)
merges <- list()
while (length(vocab) < vocab_size) {
pair <- get_most_common_pair(dataset)
nrow(pair) || break
dataset <- dataset |> merge_pair(pair)
new_token <- str_flatten(pair)
merges[[length(merges) + 1]] <- pair
vocab[[length(vocab) + 1]] <- new_token
}
list(vocab = vocab, merges = merges)
}
This R function makes the size limit explicit: it starts with a character vocabulary, adds each merged pair as a new token, and returns both the vocabulary and the merges needed by the tokenizer. section 14.2.2
Vocabulary size also shapes a model. For IMDb sentiment classification, a Keras `TextVectorization` layer learns a 20,000-word vocabulary from training data and produces a 20,000-position multi-hot vector: each position tracks whether a term appears. A linear classifier then has one weight per vocabulary position, plus a bias. Adding bigrams lets adjacent word pairs compete with single words for space in a larger, 30,000-term vocabulary. section 14.4.1 section 14.4.2
For text generation, the vocabulary defines the possible next-token outputs. Predicting one token at a time avoids treating every possible sentence as a separate output: a 20,000-word vocabulary requires 20,000 next-token outputs, while four-word sequences already have 20,000⁴ possibilities. A character-level Shakespeare model instead learns 67 vocabulary entries and predicts a probability for each character. section 15.1 section 15.1.1
A vocabulary need not be built by hand. `TextVectorization` learns one with `adapt()`; for larger-scale preprocessing, a SentencePiece tokenizer can load a prepared 32,000-term subword vocabulary and convert between text and integer tokens. section 14.4.1 section 16.2
