liveBook · Manning

SubWordTokenizer

As explained in Deep Learning with R, Third Edition

SubWordTokenizer is a book-specific tokenizer that applies learned subword merge rules to character-split text.

SubWordTokenizer applies learned byte-pair merge rules to character-split text, producing whole words, partial words, or individual characters. It combines compact vocabularies with coverage for rare words. section 14.2.2

Moby Dick tokenizer

A 2,000-token vocabulary built from Moby Dick can tokenize every word without [UNK]. Common words remain whole, while rare words fall back to smaller subwords or characters. section 14.2.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains vocabulary learning and merge application with runnable R functions, stopping short of formal derivations.
Examples
Uses a toy sentence, progressively merged symbols, and a Moby Dick vocabulary and test sentence.
What you can do after
Build it from scratch in R, train merge rules, tokenize new strings, and detokenize resulting token indices.
How it connects
Connects byte-pair encoding with character tokenization, word tokenization, vocabularies, and unknown-token handling.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.