Subword tokenization
Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
Natural Language Processing in Action, Second Edition
A subword is a word fragment, character sequence, or complete word treated as a unit in subword-based language modeling.
- How it is used
- Uses character-derived units to balance vocabulary size, sequence coverage, and generalization in deep-learning NLP pipelines.
- How deep it goes
- Explains token construction, character n-gram statistics, vocabulary coverage, and embedding composition, but does not specify a complete tokenizer algorithm.
- Examples
- Demonstrates character bigram counts with CountVectorizer and composes fastText representations for rare or unseen words.
- What you can do after
- Apply it via CountVectorizer or fastText to inspect character fragments and improve representations for rare and unseen words.
Read how Natural Language Processing in Action, Second Edition explains subword tokenization →
Deep Learning with R, Third Edition
A subword token is a token representing a complete word, a word fragment, or a character-level fallback unit.
- How it is used
- Preprocesses text into integer-token sequences for text classification, pretrained Transformers, and language-model training.
- How deep it goes
- Works through frequent-pair merging and R tokenizer code, then uses pretrained tokenizers without detailing their internal implementations.
- Examples
- Uses repeated fox sentences for pair merges, Moby-Dick for vocabulary comparisons, and RoBERTa and SentencePiece tokenizers for pretrained workflows.
- What you can do after
- Build it from scratch with byte-pair merges, or apply it via KerasHub tokenizers to produce integer sequences.
Read how Deep Learning with R, Third Edition explains subword tokenization →
Learn Generative AI with PyTorch
A partial word unit produced by subword tokenization, smaller than a whole word but larger than a character.
- How it is used
- Prepares English and French phrases as tokens for vocabulary indexing and Transformer translation.
- How deep it goes
- Explains tokenization tradeoffs and works through pretrained-tokenizer outputs and token-index mapping, but defers detailed BPE workings.
- Examples
- Uses split English and French phrases, XLMTokenizer output, and dictionaries that convert tokens to indexes and back.
- What you can do after
- Apply it via XLMTokenizer to tokenize translation pairs and map their tokens to integer sequences.
Read how Learn Generative AI with PyTorch explains subword tokenization →
Deep Learning with Python, Third Edition
A tokenization method that splits text into units smaller than words but larger than characters to balance vocabulary size and sequence length.
- How it is used
- Balances sequence length and vocabulary size when turning text into integer inputs for models.
- How deep it goes
- Works through frequency-weighted byte-pair merges and a tokenizer vocabulary, then compares resulting sequence lengths.
- Examples
- Uses repeated fox sentences for merges, a sample line for token counts, and RoBERTa for pretrained-model use.
- What you can do after
- Build it from scratch with byte-pair merges, or apply it via KerasHub’s pretrained RoBERTa tokenizer.
Read how Deep Learning with Python, Third Edition explains subword tokenization →
How Large Language Models Work
A subword is a word fragment that lies between a complete word and an individual character and can carry part of a word's meaning.
- How it is used
- Supports language-model tokenization by representing familiar words and unfamiliar words as reusable fragments.
- How deep it goes
- Explains subword meaning, BPE construction, vocabulary tradeoffs, and token-relationship limitations, but stops short of model-training internals.
- Examples
- Uses schoolhouse, thoughtful, and repeated merging of the frequent letter pair ng to demonstrate subword segmentation.
- What you can do after
- Evaluate how tokenization represents unfamiliar words and why vocabulary choices affect model coverage, storage, and computation.
Read how How Large Language Models Work explains subword tokenization →
LLMs in Production
A subword is a tokenization unit corresponding to part of a word rather than necessarily a complete word.
- How it is used
- Determines the textual units and numerical inputs available to an LLM during embedding, inference, and arithmetic tasks.
- How deep it goes
- Explains splitting and numbering, contrasts frequency grouping with per-digit handling, and demonstrates tokenizer training without covering model internals.
- Examples
- Uses arithmetic prompts, digit-level Llama tokenization, token ID output, and a Python ByteLevel BPE training listing.
- What you can do after
- Build it from scratch with Python tokenizers by training a ByteLevel BPE tokenizer on local text files.
Build a Large Language Model (From Scratch)
A token fragment smaller than a full word, used by subword tokenizers to represent rare or unseen words.
- How it is used
- Uses subword units to tokenize rare or unseen words before converting text into token IDs and embeddings.
- How deep it goes
- Defines BPE’s iterative merging process and demonstrates encoding, decoding, and unknown-word handling, but defers a detailed implementation.
- Examples
- Demonstrates “someunknownPlace,” “Akwirw ier,” character fallback, and tiktoken encode–decode round trips.
- What you can do after
- Apply it via tiktoken to encode unfamiliar words, inspect token IDs, and verify decoded text.
Read how Build a Large Language Model (From Scratch) explains subword tokenization →
Related concepts
- tokenizationkind of · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
- byte pair encodingincludes · Byte-pair encoding is a subword tokenization method that iteratively merges frequent adjacent symbol pairs to build a vocabulary of larger units.
- SentencePieceused by · SentencePiece is a language-independent software library that learns and applies subword tokenization directly to raw text without requiring whitespace-based pretokenization.
- out-of-vocabulary wordaddresses · An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s available vocabulary and may be represented by a special unknown-token marker.
- FastTextused by · FastText is a word-representation model that composes word vectors from character n-grams and whole-word information, including subword representations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.