liveBook · Manning

Byte pair encoding

As explained in Deep Learning with Python, Third Edition

A lossless-compression algorithm adapted here to build subword vocabularies by merging frequent symbol pairs.

Byte-pair encoding adapts lossless compression to tokenization: start with characters, repeatedly merge the most frequent adjacent symbol pair, and grow a vocabulary of subwords. section 14.2.2

A worked merge

Split words such as “brown” into “b r o w n”, count adjacent pairs across word frequencies, and merge the most frequent pair. “o w” becomes “ow”; later merges can turn it into “low”. Repeating the process produces entries such as “the”, “brown”, and “fox”. section 14.2.2

How Deep Learning with Python, Third Edition covers it

How deep it goes
Explains preprocessing, weighted pair counting, repeated merging, and vocabulary sizing, without detailing the complete tokenizer implementation.
Examples
Uses a toy corpus, character-split word counts, successive merges, and a 2,000-entry vocabulary trained on Moby-Dick.
What it assumes
Starts from character-level symbols and word counts, making the procedure accessible without assuming a pretrained tokenizer.
What you can do after
Build a subword vocabulary and tokenizer by implementing character splitting, pair counting, and frequency-based merging.
How it connects
Connects subword tokenization with character tokenization, word tokenization, and lossless compression.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 11 books explain byte pair encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.