liveBook · Manning

Subword tokenization

As explained in Deep Learning with R, Third Edition

A subword token is a token representing a complete word, a word fragment, or a character-level fallback unit.

Subword tokenization splits text into units that can be whole words, word fragments, or individual characters. It aims to keep sequences shorter than character tokenization without requiring a separate vocabulary entry for every word. section 14.2.2 section 14.2.1

Splitting a sentence

A subword tokenizer trained on Moby-Dick can split “Ishmael” into “ish”, “ma”, and “el”, while keeping words such as “some” and “years” together. It splits “precisely” into “precise” and “ly”. The pieces depend on the learned vocabulary. section 14.2.2

The balance matters: character tokens can represent many inputs with a small vocabulary, but produce long sequences. Word tokens compress sequences, but a restricted vocabulary can replace unfamiliar words with [UNK], losing information. Subwords reuse common pieces to offer a middle ground. section 14.2.1 section 14.2.2

Tokenization turns text into integer data in three steps: standardize the text, split it into tokens, and index each token with an integer ID. A subword tokenizer uses its learned vocabulary and merge rules during splitting. section 14.6 section 14.2.2

One way to learn subwords is byte-pair encoding (BPE). Start with characters, count adjacent token pairs within words, and merge the most frequent pair into a new token. Repeat the count and merge on the updated tokens until the vocabulary reaches its target size or no pairs remain. Keep the ordered merges so the tokenizer can apply them to new text. section 14.2.2

A first BPE merge

In a small dataset containing “brown” three times and “slow” once, the adjacent pair “o”, “w” appears four times. Merging it creates the token “ow”; a later merge can combine “l” and “ow” into “low”. section 14.2.2

Listing 14.5 Computing a byte-pair encoding vocabulary
compute_sub_word_vocabulary <- function(dataset, vocab_size) {
  dataset <- split_chars(dataset)
  vocab <- compute_char_vocabulary(dataset)
  merges <- list()
  while (length(vocab) < vocab_size) {
    pair <- get_most_common_pair(dataset)
    nrow(pair) || break
    dataset <- dataset |> merge_pair(pair)
    new_token <- str_flatten(pair)
    merges[[length(merges) + 1]] <- pair
    vocab[[length(vocab) + 1]] <- new_token
  }
  list(vocab = vocab, merges = merges)
}

See the R loop that builds a character vocabulary, adds frequent merged pairs, and records their order. section 14.2.2

For practical preprocessing, SentencePiece provides a fast subword tokenizer with detokenization. KerasHub can load a pretrained vocabulary as a tokenizer layer; the resulting integer sequences can be used to prepare inputs for language-model training. section 16.2

Listing 16.2 Downloading a vocabulary and instantiating a tokenizer
py_require("keras_hub")
keras_hub <- import("keras_hub")

vocabulary_file <- get_file(
  origin = "https://hf.co/mattdangerw/spiece/resolve/main/vocabulary.proto"
)
tokenizer <- keras_hub$tokenizers$SentencePieceTokenizer(vocabulary_file)

Load a pretrained SentencePiece vocabulary and create the KerasHub tokenizer in R. section 16.2

(tokenized <- tokenizer$tokenize("The quick brown fox."))

Array([  450,  4996, 17354,  1701, 29916, 29889], dtype = int32)

tokenizer$detokenize(tokenized)

[1] "The quick brown fox."

See a sentence converted to integer IDs and detokenized back to text. section 16.2

Vocabulary size is a practical choice rather than a fixed rule. For example, RoBERTa uses a 50,000-term subword vocabulary to avoid both excessively long character sequences and a massive word vocabulary. section 15.4.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Builds a byte-pair encoding vocabulary and tokenizer, then uses pretrained tokenizer assets in larger models.
Examples
Uses Moby-Dick sentence fragments, frequent pairs in a fox dataset, and SentencePiece round-trip tokenization.
What you can do after
Build a basic BPE tokenizer from scratch or apply a pretrained SentencePiece tokenizer via KerasHub.
How it connects
Connects byte-pair encoding with character and word tokenization, vocabulary size, unknown tokens, and embeddings.

Also explained in

Compare how 7 books explain subword tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.