A text model needs numbers, not raw strings. Tokenization turns text into token IDs in three steps: standardize the input, split it into tokens, then look up each token in a vocabulary. Word splitting produces tokens corresponding mainly to words, but punctuation can be a token too. section 14.2
From a sentence to IDs
In R, a small word-tokenization pipeline looks like this: library(stringr) text <- "The quick brown fox jumped over the lazy dog." vocabulary <- c("the", "quick", "brown", "fox", "jumped", "over", "dog", ".") words <- str_split(str_to_lower(text), boundary("word", skip_word_none = FALSE)) |> unlist() |> str_subset("\\S") match(words, vocabulary, nomatch = 0L) # 1 2 3 4 5 6 1 0 7 8 Lowercasing lets both instances of “the” share an ID. “lazy” receives 0 because it is absent from the vocabulary. The period has its own ID. section 14.2 section 14.2.1
A tokenizer packages those steps into a repeatable mapping. Its vocabulary can be built by counting tokens in training text and keeping the most common ones. A size limit controls how many token IDs—and, later, model parameters—are needed. Tokens outside the vocabulary map to an unknown-token value rather than receiving new IDs. section 14.2.1
Listing 14.3 A basic word-level tokenizer
new_word_tokenizer <- function(vocabulary, nomatch = "[UNK]") {
self <- new.env(parent = emptyenv())
attr(self, "class") <- "WordTokenizer"
vocabulary; nomatch; #1
self$standardize <- function(string) {
tolower(string)
}
self$split <- function(inputs) {
split_words(inputs)
}
self$index <- function(tokens) {
match(tokens, vocabulary, nomatch = 0)
}
self$tokenize <- function(string) { #2
string |>
self$standardize() |>
self$split() |>
self$index()
}
self$detokenize <- function(indices) { #3
indices[indices == 0] <- NA
matches <- vocabulary[indices]
matches[is.na(matches)] <- nomatch
matches
}
self
}
Inspect a word tokenizer with separate standardize, split, index, tokenize, and detokenize functions. section 14.2.1
Token size is a practical choice. Character tokenization needs a small vocabulary but makes long sequences; word tokenization shortens sequences but can require a large vocabulary and lose uncommon words. In one full-text comparison, characters needed 89 vocabulary terms, while indexing every word would have required about 17,000. Subword tokenization offers another choice when shorter sequences and coverage of rare words both matter. section 14.2.1 section 14.2.2
Choose the split to suit the task. For movie-review sentiment, word tokens feed a multi-hot bag-of-words classifier: a bag of individual characters would reveal little about a review’s content. For next-character prediction, character tokens preserve an ordered sequence, with each label offset by one character. Tokenization runs before numeric inputs reach a GPU, so preprocessing speed can also matter. section 14.4.1 section 15.1.1
