Standardization normalizes text before tokenization so irrelevant encoding differences do not create separate representations. Common transformations include lowercasing, punctuation removal, and accent normalization. section 14.2 section 14.6
Making equivalent text more similar
“Sunset came. I stared at the México sky. Isn’t nature splendid?” becomes “sunset came i stared at the méxico sky isnt nature splendid” after lowercasing and punctuation removal. Removing accent marks would make “México” and “Mexico” match as well. section 14.2
A tokenizer applies standardization as the first part of a simple preprocessing pipeline:
- Standardize the input, such as by converting characters to lowercase. section 14.2.1
- Split the standardized text into characters or words. section 14.2.1
- Index tokens against a vocabulary, assigning unknown terms a special index. section 14.2
Listing 14.1 A basic character-level tokenizer
new_char_tokenizer <- function(vocabulary, nomatch = "[UNK]") {
self <- new.env(parent = emptyenv())
attr(self, "class") <- "CharTokenizer"
self$vocabulary <- vocabulary
self$nomatch <- nomatch
self$standardize <- function(strings) {
str_to_lower(strings)
}
self$split <- function(strings) {
split_chars(strings)
}
self$index <- function(tokens) {
match(tokens, self$vocabulary, nomatch = 0L)
}
self$tokenize <- function(strings) { #1
strings |>
self$standardize() |>
self$split() |>
self$index()
}
self$detokenize <- function(indices) { #2
indices[indices == 0] <- NA
matches <- self$vocabulary[indices]
matches[is.na(matches)] <- self$nomatch
matches
}
self
}
Inspect a complete character-level tokenizer that lowercases input before splitting and indexing it. section 14.2.1
