liveBook · Manning

Byte pair encoding

As explained in Deep Learning with R, Third Edition

Byte-pair encoding is an iterative algorithm that builds a vocabulary by repeatedly merging the most frequent adjacent symbol pairs.

Builds a compact subword vocabulary by starting with characters, counting adjacent token pairs, and repeatedly merging the most frequent pair. This reduces sequence length while retaining a small vocabulary. section 14.2.2

Merging a toy corpus

For “the quick brown fox”, “the slow brown fox”, and “the quick brown foxhound”, the pair “o w” occurs most often. Merging it creates “ow”; later merges can turn “ow” plus “l” into “low”. The process continues until the desired vocabulary size is reached. section 14.2.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains pair counting, iterative merging, vocabulary construction, and tokenizer application without deriving algorithmic internals.
Examples
Uses a toy corpus containing quick, brown, slow, fox, and foxhound, plus a Moby-Dick vocabulary and detokenization example.
What it assumes
Assumes basic text tokenization concepts while teaching character splitting, pair counting, merging, and indexing concretely.
What you can do after
Builds an R byte-pair encoding vocabulary and tokenizer from a dataset and chosen vocabulary size.
How it connects
Connects tokenization with vocabulary construction, subword tokenization, compression, and unknown-token handling.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 11 books explain byte pair encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.