liveBook · Manning

Byte pair encoding

As explained in Fight Fraud with Machine Learning

Byte-pair encoding is a subword tokenization method that builds a vocabulary by repeatedly merging frequent adjacent symbol pairs.

Byte-pair encoding (BPE) tokenizes text as subwords rather than whole words. It builds a vocabulary by repeatedly merging frequent adjacent symbols, placing its units between individual letters and complete words. section 5.2.1

This compromise controls two competing costs: word-level tokenization creates a very large vocabulary, while alphabet-level tokenization keeps the vocabulary small but makes each sentence much longer in tokens. GPT uses BPE to encode text before processing tokens. section 5.2.1

The tokenization choices can be compared directly:

  • Words: fewer tokens per sentence, but many distinct vocabulary entries. section 5.2.1
  • Alphabets: a limited vocabulary of 26 symbols, but more tokens per sentence. section 5.2.1
  • Subwords through BPE: a middle ground between those extremes. section 5.2.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Defines the subword compromise but defers the detailed merge procedure to an external resource.
Examples
Compares whole words with individual alphabet characters as alternative token units.
What you can do after
Evaluate/understand why subword tokens balance vocabulary size against tokens per sentence.
How it connects
Connects byte-pair encoding to one-hot encoding, tokens, vocabulary size, GPT, and embeddings.

Also explained in

Compare how 14 books explain byte pair encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.