liveBook · Manning

Subword tokenization

Also called subword-level tokenization

Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.

How 7 Manning books explain it

Natural Language Processing in Action, Second Edition

A subword is a word fragment, character sequence, or complete word treated as a unit in subword-based language modeling.

How it is used
Uses character-derived units to balance vocabulary size, sequence coverage, and generalization in deep-learning NLP pipelines.
How deep it goes
Explains token construction, character n-gram statistics, vocabulary coverage, and embedding composition, but does not specify a complete tokenizer algorithm.
Examples
Demonstrates character bigram counts with CountVectorizer and composes fastText representations for rare or unseen words.
What you can do after
Apply it via CountVectorizer or fastText to inspect character fragments and improve representations for rare and unseen words.

Read how Natural Language Processing in Action, Second Edition explains subword tokenization →

Deep Learning with R, Third Edition

A subword token is a token representing a complete word, a word fragment, or a character-level fallback unit.

How it is used
Preprocesses text into integer sequences for classification and language-model pretraining.
How deep it goes
Builds a byte-pair encoding vocabulary and tokenizer, then uses pretrained tokenizer assets in larger models.
Examples
Uses Moby-Dick sentence fragments, frequent pairs in a fox dataset, and SentencePiece round-trip tokenization.
What you can do after
Build a basic BPE tokenizer from scratch or apply a pretrained SentencePiece tokenizer via KerasHub.

Read how Deep Learning with R, Third Edition explains subword tokenization →

Learn Generative AI with PyTorch

A partial word unit produced by subword tokenization, smaller than a whole word but larger than a character.

How it is used
Prepares English–French phrases as indexed token sequences for Transformer translation training.
How deep it goes
Explains vocabulary tradeoffs and token indexing with pretrained examples, while leaving detailed BPE workings out.
Examples
Uses English and French translation phrases and a short GPT-2 BPE encoding and decoding example.
What you can do after
Apply it via XLMTokenizer to tokenize bilingual phrases and map tokens to indexes.

Read how Learn Generative AI with PyTorch explains subword tokenization →

Deep Learning with Python, Third Edition

A tokenization method that splits text into units smaller than words but larger than characters to balance vocabulary size and sequence length.

How it is used
Uses subword tokenization to balance vocabulary size and sequence length for text models, including pretrained Transformers.
How deep it goes
Works through byte-pair counts and merges, vocabulary construction, and tokenization, then compares a pretrained tokenizer.
Examples
Compares token counts for a sample sentence and traces merges on a small dataset containing “brown” and “slow”.
What you can do after
Builds a byte-pair tokenizer from scratch and applies a pretrained RoBERTa tokenizer through KerasHub.

Read how Deep Learning with Python, Third Edition explains subword tokenization →

How Large Language Models Work

A subword is a word fragment that lies between a complete word and an individual character and can carry part of a word's meaning.

How it is used
Supports language-model tokenization by representing familiar words and unfamiliar words as reusable fragments.
How deep it goes
Explains subword meaning, BPE construction, vocabulary tradeoffs, and token-relationship limitations, but stops short of model-training internals.
Examples
Uses schoolhouse, thoughtful, and repeated merging of the frequent letter pair ng to demonstrate subword segmentation.
What you can do after
Evaluate how tokenization represents unfamiliar words and why vocabulary choices affect model coverage, storage, and computation.

Read how How Large Language Models Work explains subword tokenization →

LLMs in Production

A subword is a tokenization unit corresponding to part of a word rather than necessarily a complete word.

How it is used
Determines the textual units and numerical inputs available to an LLM during embedding, inference, and arithmetic tasks.
How deep it goes
Explains splitting and numbering, contrasts frequency grouping with per-digit handling, and demonstrates tokenizer training without covering model internals.
Examples
Uses arithmetic prompts, digit-level Llama tokenization, token ID output, and a Python ByteLevel BPE training listing.
What you can do after
Build it from scratch with Python tokenizers by training a ByteLevel BPE tokenizer on local text files.

Read how LLMs in Production explains subword tokenization →

Build a Large Language Model (From Scratch)

A token fragment smaller than a full word, used by subword tokenizers to represent rare or unseen words.

How it is used
Uses subword units to tokenize rare or unseen words before converting text into token IDs and embeddings.
How deep it goes
Defines BPE’s iterative merging process and demonstrates encoding, decoding, and unknown-word handling, but defers a detailed implementation.
Examples
Demonstrates “someunknownPlace,” “Akwirw ier,” character fallback, and tiktoken encode–decode round trips.
What you can do after
Apply it via tiktoken to encode unfamiliar words, inspect token IDs, and verify decoded text.

Read how Build a Large Language Model (From Scratch) explains subword tokenization →

Also covered in

What to know first

Related concepts

Questions about subword tokenization

What is subword tokenization?

Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.

Which Manning books explain subword tokenization?

Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition, Learn Generative AI with PyTorch, Deep Learning with Python, Third Edition, How Large Language Models Work, LLMs in Production and Build a Large Language Model (From Scratch).

What should I know before learning subword tokenization?

Manning books that teach subword tokenization usually introduce language model, vocabulary, natural language processing and BERT first.

What kinds of subword tokenization are there?

Kinds of subword tokenization include byte pair encoding.

What is subword tokenization a kind of?

Subword tokenization is a kind of tokenization. Tokenization is the process of dividing text into discrete units called tokens for computational processing.

What is subword tokenization also called?

Subword tokenization is also called subword-level tokenization.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.