Subword tokenization treats character sequences, word fragments, syllables, or complete words as language-modeling units. It occupies a useful middle ground: unlike word tokenization, it can represent unfamiliar words; unlike character tokenization, it can preserve frequent, meaningful chunks. Subword vocabularies therefore balance sequence length, vocabulary size, and generalization. section 2.2 section 2.2.1
A subword tokenizer builds units from neighboring characters rather than relying only on spaces or punctuation:
- Count character n-grams and identify sequences that frequently occur together, such as common parts of words or compounds. section 2.2.1
- Keep individual characters in the vocabulary so unfamiliar words can be decomposed, provided they contain no unseen characters. section 2.2.1
- Use frequent whole words or word pieces when their statistics support them, while retaining smaller pieces for rare forms. section 2.2 section 2.2.1
Inspecting character n-grams in Python
A character analyzer makes the intermediate vocabulary visible: CountVectorizer(ngram_range=(1, 2), analyzer='char') learns single characters and adjacent pairs from texts. Transforming the texts into a frequency table shows which fragments dominate; filtering by length then isolates two-character candidates such as “en”, “an”, and “uc”. section 2.2.1
The same idea supports embeddings. fastText represents n-character grams—including words, misspellings, partial words, and single characters—and can compose a rare or unseen word from familiar pieces. For example, a scarce “Superwoman” can draw on vectors for “super” and “woman,” reducing reliance on a generic out-of-vocabulary vector. This improves coverage, but pretrained model quality still varies with the language and available training corpus. section 6.4.2
