liveBook · Manning

Smiles pair encoding

As explained in Build AI Drug Discovery Pipelines

SMILES Pair Encoding (SmilesPE) is a subword tokenization method that learns frequent token pairs or molecular substrings in SMILES strings and encodes them as merged tokens.

SMILES Pair Encoding (SmilesPE or SPE) tokenizes molecular SMILES strings by repeatedly merging frequent adjacent token pairs into single tokens. Compared with character-level tokenization, it captures recurring chemical motifs, shortens sequences, and gives generative models a stronger inductive bias. section 10.4.4 section 10.5

A typical use in molecular generation follows this workflow:

  • Start with basic SMILES tokens, such as individual characters or an initial vocabulary. section 10.4.4
  • Count adjacent token pairs and merge the most frequent pair into a new vocabulary token. section 10.4.4
  • Repeat the merging process to represent recurring molecular substrings as single tokens. section 10.4.4 section 10.5
  • Feed the shorter token sequences into an encoder–decoder model such as the VAE-CYC molecular autoencoder. section 10.4.5

Why shorter molecular sequences help

A character-level tokenizer treats c1ccccc1 one character at a time, requiring the model to learn that the sequence represents a benzene ring. SmilesPE can instead learn frequent multi-token substrings, reducing sequence length and giving the model chemically useful structure earlier. section 10.4.4 section 10.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains iterative pair merging and representation benefits, but does not detail tokenizer training code or vocabulary-selection criteria.
Examples
Uses the SMILES string c1ccccc1 and recurring chemical motifs or functional groups as representation examples.
What it assumes
Builds from character-level tokenization and assumes familiarity with SMILES strings and sequence-based generative models.
What you can do after
Apply it to molecular sequence modeling by replacing character-level tokens with learned recurring substrings.
How it connects
Connects tokenization with inductive bias, posterior collapse, latent representations, GRUs, and variational autoencoders.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.