Subword tokenization splits text into units that can be whole words, word fragments, or individual characters. It aims to keep sequences shorter than character tokenization without requiring a separate vocabulary entry for every word. section 14.2.2 section 14.2.1
Splitting a sentence
A subword tokenizer trained on Moby-Dick can split “Ishmael” into “ish”, “ma”, and “el”, while keeping words such as “some” and “years” together. It splits “precisely” into “precise” and “ly”. The pieces depend on the learned vocabulary. section 14.2.2
The balance matters: character tokens can represent many inputs with a small vocabulary, but produce long sequences. Word tokens compress sequences, but a restricted vocabulary can replace unfamiliar words with [UNK], losing information. Subwords reuse common pieces to offer a middle ground. section 14.2.1 section 14.2.2
Tokenization turns text into integer data in three steps: standardize the text, split it into tokens, and index each token with an integer ID. A subword tokenizer uses its learned vocabulary and merge rules during splitting. section 14.6 section 14.2.2
One way to learn subwords is byte-pair encoding (BPE). Start with characters, count adjacent token pairs within words, and merge the most frequent pair into a new token. Repeat the count and merge on the updated tokens until the vocabulary reaches its target size or no pairs remain. Keep the ordered merges so the tokenizer can apply them to new text. section 14.2.2
A first BPE merge
In a small dataset containing “brown” three times and “slow” once, the adjacent pair “o”, “w” appears four times. Merging it creates the token “ow”; a later merge can combine “l” and “ow” into “low”. section 14.2.2
Listing 14.5 Computing a byte-pair encoding vocabulary
compute_sub_word_vocabulary <- function(dataset, vocab_size) {
dataset <- split_chars(dataset)
vocab <- compute_char_vocabulary(dataset)
merges <- list()
while (length(vocab) < vocab_size) {
pair <- get_most_common_pair(dataset)
nrow(pair) || break
dataset <- dataset |> merge_pair(pair)
new_token <- str_flatten(pair)
merges[[length(merges) + 1]] <- pair
vocab[[length(vocab) + 1]] <- new_token
}
list(vocab = vocab, merges = merges)
}
See the R loop that builds a character vocabulary, adds frequent merged pairs, and records their order. section 14.2.2
For practical preprocessing, SentencePiece provides a fast subword tokenizer with detokenization. KerasHub can load a pretrained vocabulary as a tokenizer layer; the resulting integer sequences can be used to prepare inputs for language-model training. section 16.2
Listing 16.2 Downloading a vocabulary and instantiating a tokenizer
py_require("keras_hub")
keras_hub <- import("keras_hub")
vocabulary_file <- get_file(
origin = "https://hf.co/mattdangerw/spiece/resolve/main/vocabulary.proto"
)
tokenizer <- keras_hub$tokenizers$SentencePieceTokenizer(vocabulary_file)
Load a pretrained SentencePiece vocabulary and create the KerasHub tokenizer in R. section 16.2
(tokenized <- tokenizer$tokenize("The quick brown fox."))
Array([ 450, 4996, 17354, 1701, 29916, 29889], dtype = int32)
tokenizer$detokenize(tokenized)
[1] "The quick brown fox."
See a sentence converted to integer IDs and detokenized back to text. section 16.2
Vocabulary size is a practical choice rather than a fixed rule. For example, RoBERTa uses a 50,000-term subword vocabulary to avoid both excessively long character sequences and a massive word vocabulary. section 15.4.2






