Byte-pair encoding (BPE) tokenizes sequences by repeatedly replacing the most frequent adjacent token pair with a newly formed token. In molecular generation, this moves beyond individual characters toward chemically meaningful motifs, producing shorter sequences that can provide a stronger inductive bias for latent-variable models and reduce posterior collapse. section 10.4.4
A chemistry-oriented BPE workflow proceeds as follows:
- Tokenize a large SMILES dataset at the atom level, keeping multi-character atoms such as [C@@H] and Br indivisible. section 10.4.4
- Initialize the vocabulary with the unique atom-level tokens. section 10.4.4
- Count adjacent token pairs, merge the most frequent pair into a new token, and replace every occurrence. section 10.4.4
- Repeat until reaching the desired vocabulary size or until no pair exceeds the frequency threshold. section 10.4.4
- Tokenize new SMILES strings using the trained vocabulary. section 10.4.4
Frequent molecular pair
If c and c occur most often as adjacent tokens, merge them into cc and add cc to the vocabulary. section 10.4.4













