SMILES Pair Encoding (SmilesPE or SPE) tokenizes molecular SMILES strings by repeatedly merging frequent adjacent token pairs into single tokens. Compared with character-level tokenization, it captures recurring chemical motifs, shortens sequences, and gives generative models a stronger inductive bias. section 10.4.4 section 10.5
A typical use in molecular generation follows this workflow:
- Start with basic SMILES tokens, such as individual characters or an initial vocabulary. section 10.4.4
- Count adjacent token pairs and merge the most frequent pair into a new vocabulary token. section 10.4.4
- Repeat the merging process to represent recurring molecular substrings as single tokens. section 10.4.4 section 10.5
- Feed the shorter token sequences into an encoder–decoder model such as the VAE-CYC molecular autoencoder. section 10.4.5
Why shorter molecular sequences help
A character-level tokenizer treats c1ccccc1 one character at a time, requiring the model to learn that the sequence represents a benzene ring. SmilesPE can instead learn frequent multi-token substrings, reducing sequence length and giving the model chemically useful structure earlier. section 10.4.4 section 10.5
