Byte pair encoding
Byte-pair encoding is a subword tokenization method that iteratively merges frequent adjacent symbol pairs to build a vocabulary of larger units.
Natural Language Processing in Action, Second Edition
Byte pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent symbol pairs to form a vocabulary.
- How it is used
- Uses BPE to keep NLP and transformer vocabularies manageable while representing text as subword tokens.
- How deep it goes
- Explains recurring character sequences and byte-level vocabulary, then counts character n-grams; stops short of implementing the iterative merge algorithm.
- Examples
- Uses CountVectorizer character counts and GPT-2 outputs such as “ new” and “ non” to expose subword pieces.
- What you can do after
- Evaluate/understand when subword tokens suit vocabulary constraints and interpret GPT-2 next-token outputs as pieces rather than necessarily whole words.
Read how Natural Language Processing in Action, Second Edition explains byte pair encoding →
Build a Large Language Model (From Scratch)
A subword tokenization method that iteratively merges frequent symbol pairs to build a vocabulary for text representation.
- How it is used
- Prepares raw text for GPT-style LLM training by producing token IDs from words, subwords, and characters.
- How deep it goes
- Explains subword fallback and demonstrates encode/decode usage, but delegates efficient algorithm implementation to tiktoken.
- Examples
- Demonstrates tiktoken with GPT-2, <|endoftext|>, an unfamiliar place name, encoded IDs, and decoded text.
- What you can do after
- Apply it via tiktoken to encode text into GPT-compatible IDs and decode those IDs back into text.
Read how Build a Large Language Model (From Scratch) explains byte pair encoding →
How Large Language Models Work
A byte-pair encoding tokenizer segments text by repeatedly merging frequent adjacent symbol pairs into tokens.
- How it is used
- Automates text segmentation into subword tokens that supply inputs to large language models.
- How deep it goes
- Explains iterative pair merging and token-count effects, but stops short of implementation code or detailed computation.
- Examples
- Uses the “ng” to “ing” merge and contrasts tokenization of “I’m running” with “I’m runnin.”
- What you can do after
- Evaluate token counts and recognize why frequent word forms, unusual spellings, and tokenizer differences can affect software behavior.
Read how How Large Language Models Work explains byte pair encoding →
LLMs in Production
Byte pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent symbol pairs into larger vocabulary units.
- How it is used
- Transforms training or fine-tuning text into model-consumable numbers while adapting token boundaries to a specific task.
- How deep it goes
- Explains subword trade-offs and training parameters but defers detailed algorithm mechanics.
- Examples
- Uses custom Python tokenizer training, token ID output, digit-level tokenization, and arithmetic-comparison results.
- What you can do after
- Build it from scratch with the tokenizers library and configure vocabulary size, frequency thresholds, and special tokens.
Learn Generative AI with PyTorch
A subword tokenization method adapted from data compression that merges frequent character pairs to represent text as tokens.
- How it is used
- Supports GPT-2 and other language models by converting text into compact subword-token sequences before embedding.
- How deep it goes
- Defines iterative frequent-pair merging and its goals, but defers detailed workings to an existing tokenizer implementation.
- Examples
- Demonstrates tokenizing “This is the original text.”, inspecting indexes, and decoding them back into text.
- What you can do after
- Apply it via the supplied Python BPE tokenizer to encode text, inspect token indexes, and decode them.
Read how Learn Generative AI with PyTorch explains byte pair encoding →
Deep Learning with R, Third Edition
Byte-pair encoding is an iterative algorithm that builds a vocabulary by repeatedly merging the most frequent adjacent symbol pairs.
- How it is used
- Compresses text into information-rich subword sequences for deep learning text models.
- How deep it goes
- Explains pair counting, iterative merging, vocabulary construction, and tokenizer application without deriving algorithmic internals.
- Examples
- Uses a toy corpus containing quick, brown, slow, fox, and foxhound, plus a Moby-Dick vocabulary and detokenization example.
- What you can do after
- Builds an R byte-pair encoding vocabulary and tokenizer from a dataset and chosen vocabulary size.
Read how Deep Learning with R, Third Edition explains byte pair encoding →
Deep Learning with Python, Third Edition
A lossless-compression algorithm adapted here to build subword vocabularies by merging frequent symbol pairs.
- How it is used
- Builds compact subword vocabularies for text tokenization and downstream text classification.
- How deep it goes
- Explains preprocessing, weighted pair counting, repeated merging, and vocabulary sizing, without detailing the complete tokenizer implementation.
- Examples
- Uses a toy corpus, character-split word counts, successive merges, and a 2,000-entry vocabulary trained on Moby-Dick.
- What you can do after
- Build a subword vocabulary and tokenizer by implementing character splitting, pair counting, and frequency-based merging.
Read how Deep Learning with Python, Third Edition explains byte pair encoding →
Transformers in Action
Byte pair encoding is a subword tokenization algorithm that iteratively merges frequent symbol pairs to form a vocabulary.
- How it is used
- Prepares raw strings as subword token IDs for autoregressive inference in decoder-only transformers.
- How deep it goes
- Explains iterative merging and an encoder pipeline, but stops short of showing the complete merge implementation.
- Examples
- Uses “inventing” with the possible units “in,” “vent,” and “ing,” plus an abstract encoder listing.
- What you can do after
- Understand how frequent-pair merges turn text into reusable subword units and model-compatible vocabulary IDs.
Read how Transformers in Action explains byte pair encoding →
Hugging Face in Action
Byte-pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent units into a vocabulary of subword tokens.
- How it is used
- Supports language-model processing by converting diverse text into manageable subword tokens.
- How deep it goes
- Defines subword tokenization and illustrates outputs, but stops short of explaining merge mechanics or implementation steps.
- Examples
- Uses “exhilarating” and “unhappiness” to demonstrate subword token sequences.
- What you can do after
- Evaluate why modern language models use subword tokens for rare, unfamiliar, and morphologically varied words.
Read how Hugging Face in Action explains byte pair encoding →
Grokking AI Algorithms, Second Edition
Byte-pair encoding is a tokenization technique that iteratively merges frequent adjacent symbol pairs to form a vocabulary of subword tokens.
- How it is used
- Explains tokenization by showing how character sequences become a vocabulary of reusable subword tokens.
- How deep it goes
- Works through repeated frequency counts and merges, then stops short of a complete tokenizer implementation.
- Examples
- Uses a training paragraph, character-marked words, frequency tables, successive merges, and a Python tokenizer listing.
- What you can do after
- Build it from scratch with a simplified Python loop that counts pairs, merges tokens, and enforces vocabulary stopping rules.
Read how Grokking AI Algorithms, Second Edition explains byte pair encoding →
Generative AI in Action
Byte-pair encoding is a reversible, lossless compression and subword-tokenization algorithm that repeatedly merges frequent symbol sequences.
- How it is used
- Supports token counting for prompt sizing, model cost planning, and performance decisions in large language model applications.
- How deep it goes
- Defines compression and tokenization behavior with examples, but stops short of explaining the merge-training procedure or mathematical internals.
- Examples
- Demonstrates English “-ing” subwords, token counting for a dog-related prompt, and reversible encoding and decoding.
- What you can do after
- Apply it via tiktoken to count, encode, and decode model-compatible token sequences.
Read how Generative AI in Action explains byte pair encoding →
Related concepts
- subword tokenizationkind of · Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
- tokenizationimplements · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
- wordpiecealternative to · WordPiece is a subword tokenization method that represents text as sequences of vocabulary units, including reusable word fragments and characters.
- tiktokenimplemented by · tiktoken is an OpenAI-developed library that efficiently encodes text using model-specific byte-pair-encoding tokenizers and supports token counting.
- SentencePiecealternative to · SentencePiece is a language-independent software library that learns and applies subword tokenization directly to raw text without requiring whitespace-based pretokenization.
- tokenizerproduces · A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to identifiers for computational processing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.