A tokenizer turns text into a sequence of token indices that a model can use. The process has three parts: standardize the text, split it into tokens, and look up each token in a vocabulary. A token might be a character, a word, or a piece of a word. section 14.2 section 14.2.2
From words to indices
In R, set `vocabulary <- c("the", "quick", "brown", "fox", "jumped", "over", "dog", ".")`, then split `"The quick brown fox jumped over the lazy dog."` into words and call `match(words, vocabulary, nomatch = 0L)`. The result is `0 2 3 4 5 6 1 0 7 8`. Both `The` and `lazy` receive 0: vocabulary lookup is case-sensitive here, and `lazy` is absent. Lowercasing first would fix the case mismatch, but not the missing word. section 14.2 section 14.2.1
Listing 14.1 A basic character-level tokenizer
new_char_tokenizer <- function(vocabulary, nomatch = "[UNK]") {
self <- new.env(parent = emptyenv())
attr(self, "class") <- "CharTokenizer"
self$vocabulary <- vocabulary
self$nomatch <- nomatch
self$standardize <- function(strings) {
str_to_lower(strings)
}
self$split <- function(strings) {
split_chars(strings)
}
self$index <- function(tokens) {
match(tokens, self$vocabulary, nomatch = 0L)
}
self$tokenize <- function(strings) { #1
strings |>
self$standardize() |>
self$split() |>
self$index()
}
self$detokenize <- function(indices) { #2
indices[indices == 0] <- NA
matches <- self$vocabulary[indices]
matches[is.na(matches)] <- self$nomatch
matches
}
self
}
The character-level implementation puts standardization, splitting, indexing, and reverse lookup in one tokenizer. Its `tokenize()` method shows the order of operations. section 14.2.1
The splitting rule changes the trade-off:
- Characters cover varied input with a small vocabulary, but produce long sequences. In a Moby Dick example, 89 character tokens cover the entire text. section 14.2.1
- Words produce shorter sequences, but need a much larger vocabulary; a 2,000-term vocabulary does not cover every word in that example. section 14.2.1
- Subwords aim for short sequences without an enormous vocabulary. Byte-pair encoding starts with characters, then repeatedly merges common adjacent token pairs. section 14.2.2
The vocabulary matters as much as the splitting rule. A pretrained model needs the tokenizer used during pretraining: different token indices would no longer point to the representations its weights learned. KerasHub can load a matching RoBERTa tokenizer with `Tokenizer$from_preset("roberta_base_en")`; its Gemma `CausalLM` preset includes a matching tokenizer. section 15.4.2 section 16.3.1
Tokenizing and reversing text
A KerasHub `SentencePieceTokenizer` loaded with a vocabulary file tokenizes `"The quick brown fox."` into six integer IDs. Calling `detokenize()` on those IDs returns `"The quick brown fox."`. The same tokenizer can run inside a `tf.data` pipeline that prepares fixed-length sequences for mini-GPT training. section 16.2
