Byte pair encoding
Byte-pair encoding is a subword tokenization method that builds a vocabulary by iteratively merging frequent adjacent symbols or tokens.
Natural Language Processing in Action, Second Edition
Byte pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent symbol pairs to form a vocabulary.
- How it is used
- Uses subword vocabularies to make text representation more compact for search and transformer pipelines.
- How deep it goes
- Explains character- and byte-based vocabulary construction and inspects pair counts, while directing readers elsewhere for GPT-2’s tokenization algorithm.
- Examples
- Inspects character-pair counts and GPT-2 next-token candidates with leading spaces.
- What you can do after
- Evaluate/understand how subword vocabulary choices affect text representation, vocabulary size, and generated token pieces.
Read how Natural Language Processing in Action, Second Edition explains byte pair encoding →
Build a Large Language Model (From Scratch)
A subword tokenization method that iteratively merges frequent symbol pairs to build a vocabulary for text representation.
- How it is used
- Prepares raw text for GPT-style LLM training by producing token IDs from words, subwords, and characters.
- How deep it goes
- Explains subword fallback and demonstrates encode/decode usage, but delegates efficient algorithm implementation to tiktoken.
- Examples
- Demonstrates tiktoken with GPT-2, <|endoftext|>, an unfamiliar place name, encoded IDs, and decoded text.
- What you can do after
- Apply it via tiktoken to encode text into GPT-compatible IDs and decode those IDs back into text.
Read how Build a Large Language Model (From Scratch) explains byte pair encoding →
How Large Language Models Work
A byte-pair encoding tokenizer segments text by repeatedly merging frequent adjacent symbol pairs into tokens.
- How it is used
- Automates text segmentation into subword tokens that supply inputs to large language models.
- How deep it goes
- Explains iterative pair merging and token-count effects, but stops short of implementation code or detailed computation.
- Examples
- Uses the “ng” to “ing” merge and contrasts tokenization of “I’m running” with “I’m runnin.”
- What you can do after
- Evaluate token counts and recognize why frequent word forms, unusual spellings, and tokenizer differences can affect software behavior.
Read how How Large Language Models Work explains byte pair encoding →
Fight Fraud with Machine Learning
Byte-pair encoding is a subword tokenization method that builds a vocabulary by repeatedly merging frequent adjacent symbol pairs.
- How it is used
- Positions subword tokenization as GPT’s compromise between word and alphabet representations.
- How deep it goes
- Defines the subword compromise but defers the detailed merge procedure to an external resource.
- Examples
- Compares whole words with individual alphabet characters as alternative token units.
- What you can do after
- Evaluate/understand why subword tokens balance vocabulary size against tokens per sentence.
Read how Fight Fraud with Machine Learning explains byte pair encoding →
LLMs in Production
Byte pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent symbol pairs into larger vocabulary units.
- How it is used
- Transforms training or fine-tuning text into model-consumable numbers while adapting token boundaries to a specific task.
- How deep it goes
- Explains subword trade-offs and training parameters but defers detailed algorithm mechanics.
- Examples
- Uses custom Python tokenizer training, token ID output, digit-level tokenization, and arithmetic-comparison results.
- What you can do after
- Build it from scratch with the tokenizers library and configure vocabulary size, frequency thresholds, and special tokens.
Learn Generative AI with PyTorch
A subword tokenization method adapted from data compression that merges frequent character pairs to represent text as tokens.
- How it is used
- Supports GPT-2 and other language models by converting text into compact subword-token sequences before embedding.
- How deep it goes
- Defines iterative frequent-pair merging and its goals, but defers detailed workings to an existing tokenizer implementation.
- Examples
- Demonstrates tokenizing “This is the original text.”, inspecting indexes, and decoding them back into text.
- What you can do after
- Apply it via the supplied Python BPE tokenizer to encode text, inspect token indexes, and decode them.
Read how Learn Generative AI with PyTorch explains byte pair encoding →
Deep Learning with R, Third Edition
Byte-pair encoding is an iterative algorithm that builds a vocabulary by repeatedly merging the most frequent adjacent symbol pairs.
- How it is used
- Compresses text into information-rich subword sequences for deep learning text models.
- How deep it goes
- Explains pair counting, iterative merging, vocabulary construction, and tokenizer application without deriving algorithmic internals.
- Examples
- Uses a toy corpus containing quick, brown, slow, fox, and foxhound, plus a Moby-Dick vocabulary and detokenization example.
- What you can do after
- Builds an R byte-pair encoding vocabulary and tokenizer from a dataset and chosen vocabulary size.
Read how Deep Learning with R, Third Edition explains byte pair encoding →
Deep Learning with Python, Third Edition
A lossless-compression algorithm adapted here to build subword vocabularies by merging frequent symbol pairs.
- How it is used
- Builds compact subword vocabularies for text tokenization and downstream text classification.
- How deep it goes
- Explains preprocessing, weighted pair counting, repeated merging, and vocabulary sizing, without detailing the complete tokenizer implementation.
- Examples
- Uses a toy corpus, character-split word counts, successive merges, and a 2,000-entry vocabulary trained on Moby-Dick.
- What you can do after
- Build a subword vocabulary and tokenizer by implementing character splitting, pair counting, and frequency-based merging.
Read how Deep Learning with Python, Third Edition explains byte pair encoding →
Build AI Drug Discovery Pipelines
Byte-pair encoding is an iterative tokenization algorithm that replaces frequent adjacent token pairs with newly formed tokens.
- How it is used
- Improves molecular sequence representations for generative models by replacing character-level tokens with chemically meaningful motifs.
- How deep it goes
- Explains atom-level initialization, repeated frequency-based merging, stopping criteria, and application to new SMILES strings.
- Examples
- Uses [C@@H], Br, c c, and the merged token cc as molecular tokenization examples.
- What you can do after
- Apply a trained vocabulary to tokenize SMILES strings after constructing it through iterative pair merging.
Read how Build AI Drug Discovery Pipelines explains byte pair encoding →
Transformers in Action
Byte pair encoding is a subword tokenization algorithm that iteratively merges frequent symbol pairs to form a vocabulary.
- How it is used
- Prepares raw strings as subword token IDs for autoregressive inference in decoder-only transformers.
- How deep it goes
- Explains iterative merging and an encoder pipeline, but stops short of showing the complete merge implementation.
- Examples
- Uses “inventing” with the possible units “in,” “vent,” and “ing,” plus an abstract encoder listing.
- What you can do after
- Understand how frequent-pair merges turn text into reusable subword units and model-compatible vocabulary IDs.
Read how Transformers in Action explains byte pair encoding →
Hugging Face in Action
Byte-pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent units into a vocabulary of subword tokens.
- How it is used
- Supports language-model processing by converting diverse text into manageable subword tokens.
- How deep it goes
- Defines subword tokenization and illustrates outputs, but stops short of explaining merge mechanics or implementation steps.
- Examples
- Uses “exhilarating” and “unhappiness” to demonstrate subword token sequences.
- What you can do after
- Evaluate why modern language models use subword tokens for rare, unfamiliar, and morphologically varied words.
Read how Hugging Face in Action explains byte pair encoding →
Grokking AI Algorithms, Second Edition
Byte-pair encoding is a tokenization technique that iteratively merges frequent adjacent symbol pairs to form a vocabulary of subword tokens.
- How it is used
- Explains tokenization by showing how character sequences become a vocabulary of reusable subword tokens.
- How deep it goes
- Works through repeated frequency counts and merges, then stops short of a complete tokenizer implementation.
- Examples
- Uses a training paragraph, character-marked words, frequency tables, successive merges, and a Python tokenizer listing.
- What you can do after
- Build it from scratch with a simplified Python loop that counts pairs, merges tokens, and enforces vocabulary stopping rules.
Read how Grokking AI Algorithms, Second Edition explains byte pair encoding →
Generative AI in Action
Byte-pair encoding is a reversible, lossless compression and subword-tokenization algorithm that repeatedly merges frequent symbol sequences.
- How it is used
- Supports token counting for prompt sizing, model cost planning, and performance decisions in large language model applications.
- How deep it goes
- Defines compression and tokenization behavior with examples, but stops short of explaining the merge-training procedure or mathematical internals.
- Examples
- Demonstrates English “-ing” subwords, token counting for a dog-related prompt, and reversible encoding and decoding.
- What you can do after
- Apply it via tiktoken to count, encode, and decode model-compatible token sequences.
Read how Generative AI in Action explains byte pair encoding →
Understanding AI
Byte Pair Encoding is an iterative tokenization algorithm that merges frequently adjacent symbols or tokens to construct a subword vocabulary.
- How it is used
- Explains how written text becomes a subword vocabulary before token IDs and embeddings are used.
- How deep it goes
- Traces character initialization, repeated frequency-based merges, vocabulary sizing, and index assignment without explaining implementation details.
- Examples
- Uses common words, “unbelievable,” character tokens, and spacing changes to illustrate token boundaries.
- What you can do after
- Evaluate how wording and spacing can alter tokenization and potentially change model outputs.
Also covered in
- Build a Text-to-Image Generator (from Scratch)
- Machines that ThinkIn early access (MEAP)
- Deep Learning with PyTorch, Second Edition
- The AI Pocket Book
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- vocabularyA vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.
- tokenizationTokenization is the process of dividing text into discrete units called tokens for computational processing.
Related concepts
- subword tokenizationkind of · Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
- tokenizationkind of · Tokenization is the process of dividing text into discrete units called tokens for computational processing.
- wordpiecealternative to · WordPiece is a subword tokenization method that segments text into units from a vocabulary, allowing words to be represented by reusable word fragments.
- SentencePiecealternative to · SentencePiece is a language-independent software library for learning and applying subword tokenization directly to raw text without whitespace-based pretokenization.
- tiktokenimplemented by · tiktoken is an OpenAI-developed library that efficiently encodes text using model-specific byte-pair-encoding tokenizers and supports token counting.
Questions about byte pair encoding
What is byte pair encoding?
Byte-pair encoding is a subword tokenization method that builds a vocabulary by iteratively merging frequent adjacent symbols or tokens.
Which Manning books explain byte pair encoding?
Natural Language Processing in Action, Second Edition, Build a Large Language Model (From Scratch), How Large Language Models Work, Fight Fraud with Machine Learning, LLMs in Production, Learn Generative AI with PyTorch, Deep Learning with R, Third Edition, Deep Learning with Python, Third Edition and 6 more.
What should I know before learning byte pair encoding?
Manning books that teach byte pair encoding usually introduce language model, token, vocabulary and tokenization first.
What is the difference between byte pair encoding and wordpiece?
Byte-pair encoding is a subword tokenization method that builds a vocabulary by iteratively merging frequent adjacent symbols or tokens. WordPiece is a subword tokenization method that segments text into units from a vocabulary, allowing words to be represented by reusable word fragments.
What is byte pair encoding a kind of?
Byte pair encoding is a kind of subword tokenization. Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
What is byte pair encoding also called?
Byte pair encoding is also called BPE.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.













