liveBook · Manning

Byte pair encoding

As explained in Build AI Drug Discovery Pipelines

Byte-pair encoding is an iterative tokenization algorithm that replaces frequent adjacent token pairs with newly formed tokens.

Byte-pair encoding (BPE) tokenizes sequences by repeatedly replacing the most frequent adjacent token pair with a newly formed token. In molecular generation, this moves beyond individual characters toward chemically meaningful motifs, producing shorter sequences that can provide a stronger inductive bias for latent-variable models and reduce posterior collapse. section 10.4.4

A chemistry-oriented BPE workflow proceeds as follows:

  • Tokenize a large SMILES dataset at the atom level, keeping multi-character atoms such as [C@@H] and Br indivisible. section 10.4.4
  • Initialize the vocabulary with the unique atom-level tokens. section 10.4.4
  • Count adjacent token pairs, merge the most frequent pair into a new token, and replace every occurrence. section 10.4.4
  • Repeat until reaching the desired vocabulary size or until no pair exceeds the frequency threshold. section 10.4.4
  • Tokenize new SMILES strings using the trained vocabulary. section 10.4.4

Frequent molecular pair

If c and c occur most often as adjacent tokens, merge them into cc and add cc to the vocabulary. section 10.4.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains atom-level initialization, repeated frequency-based merging, stopping criteria, and application to new SMILES strings.
Examples
Uses [C@@H], Br, c c, and the merged token cc as molecular tokenization examples.
What it assumes
Assumes SMILES strings and atom-level tokenization, while defining how multi-character atoms become indivisible starting tokens.
What you can do after
Apply a trained vocabulary to tokenize SMILES strings after constructing it through iterative pair merging.
How it connects
Connects tokenization with SMILES, inductive bias, latent representations, posterior collapse, encoders, and decoders.

Also explained in

Compare how 14 books explain byte pair encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.