WordPiece tokenization represents text with reusable word fragments and character sequences rather than complete words. Prefixes, suffixes, and interior pieces can transfer patterns across related words, helping a language pipeline handle unfamiliar forms and large vocabularies. Unlike word tokenizers, subword tokenizers must be trained over a corpus until their vocabulary converges. section 2.2 section 2.2.1
WordPiece is useful when a pipeline needs to:
- Reuse fragments across related words, including prefixes, suffixes, and interior syllables. section 2.2
- Represent new words through smaller pieces instead of discarding them as unknown whole-word entries. section 2.2
- Compress a large natural-language vocabulary into a more computation-friendly token inventory. section 2.2 section 2.2.1
From fragments to transformer inputs
A practical exploration begins by fitting a character n-gram CountVectorizer to text, inspecting frequent one- and two-character fragments, and comparing their counts. WordPiece belongs to the same broader subword-tokenization family, but its fitted vocabulary is used directly as model tokens. In BERT, sequences add [CLS] and [SEP] tokens, combine token, segment, and positional embeddings, and randomly mask 15% of WordPiece tokens during pretraining. section 9.3.1
