An out-of-vocabulary word is a token absent from a tokenizer’s retained vocabulary. Instead of failing, the tokenizer assigns it index 0 and later represents that index with the special token "[UNK]". This lets the preprocessing pipeline encode terms that appear only in new or test data. section 14.2
The fallback is especially common with word-level tokenization, where vocabulary size is deliberately limited to frequent terms. Character-level tokenization encounters fewer unknown units, while word-level tokenization produces shorter sequences but can omit many words. section 14.2.1
A missing word in a small vocabulary
With the vocabulary ["the", "quick", "brown", "fox", "jumped", "over", "dog", "."], the sentence “The quick brown fox jumped over the lazy dog.” becomes indices 0, 2, 3, 4, 5, 6, 1, 0, 7, 8. The first 0 represents “The” before standardization, and the second represents “lazy”; both can be detokenized as "[UNK]". section 14.2
Listing 14.1 A basic character-level tokenizer
new_char_tokenizer <- function(vocabulary, nomatch = "[UNK]") {
self <- new.env(parent = emptyenv())
attr(self, "class") <- "CharTokenizer"
self$vocabulary <- vocabulary
self$nomatch <- nomatch
self$standardize <- function(strings) {
str_to_lower(strings)
}
self$split <- function(strings) {
split_chars(strings)
}
self$index <- function(tokens) {
match(tokens, self$vocabulary, nomatch = 0L)
}
self$tokenize <- function(strings) { #1
strings |>
self$standardize() |>
self$split() |>
self$index()
}
self$detokenize <- function(indices) { #2
indices[indices == 0] <- NA
matches <- self$vocabulary[indices]
matches[is.na(matches)] <- self$nomatch
matches
}
self
}
Inspect the tokenizer implementation: unmatched tokens receive index 0, and detokenization replaces missing matches with "[UNK]". section 14.2.1
Unknown words in Moby Dick
A 2,000-term vocabulary tokenizes “Call me Ishmael. Some years ago--never mind how long precisely.” with 0 for “Ishmael”; detokenization renders it as "[UNK]".
