Subword tokenization
Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
Natural Language Processing in Action, Second Edition
A subword is a word fragment, character sequence, or complete word treated as a unit in subword-based language modeling.
- How it is used
- Uses character-derived units to balance vocabulary size, sequence coverage, and generalization in deep-learning NLP pipelines.
- How deep it goes
- Explains token construction, character n-gram statistics, vocabulary coverage, and embedding composition, but does not specify a complete tokenizer algorithm.
- Examples
- Demonstrates character bigram counts with CountVectorizer and composes fastText representations for rare or unseen words.
- What you can do after
- Apply it via CountVectorizer or fastText to inspect character fragments and improve representations for rare and unseen words.
Read how Natural Language Processing in Action, Second Edition explains subword tokenization →
Deep Learning with R, Third Edition
A subword token is a token representing a complete word, a word fragment, or a character-level fallback unit.
- How it is used
- Preprocesses text into integer sequences for classification and language-model pretraining.
- How deep it goes
- Builds a byte-pair encoding vocabulary and tokenizer, then uses pretrained tokenizer assets in larger models.
- Examples
- Uses Moby-Dick sentence fragments, frequent pairs in a fox dataset, and SentencePiece round-trip tokenization.
- What you can do after
- Build a basic BPE tokenizer from scratch or apply a pretrained SentencePiece tokenizer via KerasHub.
Read how Deep Learning with R, Third Edition explains subword tokenization →
Learn Generative AI with PyTorch
A partial word unit produced by subword tokenization, smaller than a whole word but larger than a character.
- How it is used
- Prepares English–French phrases as indexed token sequences for Transformer translation training.
- How deep it goes
- Explains vocabulary tradeoffs and token indexing with pretrained examples, while leaving detailed BPE workings out.
- Examples
- Uses English and French translation phrases and a short GPT-2 BPE encoding and decoding example.
- What you can do after
- Apply it via XLMTokenizer to tokenize bilingual phrases and map tokens to indexes.
Read how Learn Generative AI with PyTorch explains subword tokenization →
Deep Learning with Python, Third Edition
A tokenization method that splits text into units smaller than words but larger than characters to balance vocabulary size and sequence length.
- How it is used
- Uses subword tokenization to balance vocabulary size and sequence length for text models, including pretrained Transformers.
- How deep it goes
- Works through byte-pair counts and merges, vocabulary construction, and tokenization, then compares a pretrained tokenizer.
- Examples
- Compares token counts for a sample sentence and traces merges on a small dataset containing “brown” and “slow”.
- What you can do after
- Builds a byte-pair tokenizer from scratch and applies a pretrained RoBERTa tokenizer through KerasHub.
Read how Deep Learning with Python, Third Edition explains subword tokenization →
How Large Language Models Work
A subword is a word fragment that lies between a complete word and an individual character and can carry part of a word's meaning.
- How it is used
- Supports language-model tokenization by representing familiar words and unfamiliar words as reusable fragments.
- How deep it goes
- Explains subword meaning, BPE construction, vocabulary tradeoffs, and token-relationship limitations, but stops short of model-training internals.
- Examples
- Uses schoolhouse, thoughtful, and repeated merging of the frequent letter pair ng to demonstrate subword segmentation.
- What you can do after
- Evaluate how tokenization represents unfamiliar words and why vocabulary choices affect model coverage, storage, and computation.
Read how How Large Language Models Work explains subword tokenization →
LLMs in Production
A subword is a tokenization unit corresponding to part of a word rather than necessarily a complete word.
- How it is used
- Determines the textual units and numerical inputs available to an LLM during embedding, inference, and arithmetic tasks.
- How deep it goes
- Explains splitting and numbering, contrasts frequency grouping with per-digit handling, and demonstrates tokenizer training without covering model internals.
- Examples
- Uses arithmetic prompts, digit-level Llama tokenization, token ID output, and a Python ByteLevel BPE training listing.
- What you can do after
- Build it from scratch with Python tokenizers by training a ByteLevel BPE tokenizer on local text files.
Build a Large Language Model (From Scratch)
A token fragment smaller than a full word, used by subword tokenizers to represent rare or unseen words.
- How it is used
- Uses subword units to tokenize rare or unseen words before converting text into token IDs and embeddings.
- How deep it goes
- Defines BPE’s iterative merging process and demonstrates encoding, decoding, and unknown-word handling, but defers a detailed implementation.
- Examples
- Demonstrates “someunknownPlace,” “Akwirw ier,” character fallback, and tiktoken encode–decode round trips.
- What you can do after
- Apply it via tiktoken to encode unfamiliar words, inspect token IDs, and verify decoded text.
Read how Build a Large Language Model (From Scratch) explains subword tokenization →
Also covered in
- Hugging Face in Action
- Machines that ThinkIn early access (MEAP)
- Starting Data Analytics with Generative AI and Python
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- vocabularyA vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
Related concepts
- byte pair encodingincludes · Byte-pair encoding is a subword tokenization method that builds a vocabulary by iteratively merging frequent adjacent symbols or tokens.
- tokenizationkind of · Tokenization is the process of dividing text into discrete units called tokens for computational processing.
- SentencePieceused by · SentencePiece is a language-independent software library for learning and applying subword tokenization directly to raw text without whitespace-based pretokenization.
- out-of-vocabulary wordaddresses · An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker.
- data tokenizationkind of · Data tokenization is a data-protection technique that replaces sensitive data with non-sensitive substitute tokens linked to the original data through a controlled mapping.
- FastTextused by · FastText is a word-representation model that learns word vectors using character n-grams and composes representations from subword information.
Questions about subword tokenization
What is subword tokenization?
Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
Which Manning books explain subword tokenization?
Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition, Learn Generative AI with PyTorch, Deep Learning with Python, Third Edition, How Large Language Models Work, LLMs in Production and Build a Large Language Model (From Scratch).
What should I know before learning subword tokenization?
Manning books that teach subword tokenization usually introduce language model, vocabulary, natural language processing and BERT first.
What kinds of subword tokenization are there?
Kinds of subword tokenization include byte pair encoding.
What is subword tokenization a kind of?
Subword tokenization is a kind of tokenization. Tokenization is the process of dividing text into discrete units called tokens for computational processing.
What is subword tokenization also called?
Subword tokenization is also called subword-level tokenization.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






