liveBook · Manning

Tokenization

As explained in Build AI Drug Discovery Pipelines

Tokenization is the conversion of structured input into a sequence of discrete tokens for computational processing.

Tokenization converts a SMILES string into a sequence of discrete numerical tokens so a neural network can process molecular structures. Character-level tokenization treats each character as a token, while a vocabulary maps tokens to integer indices. section 10.2.2

A practical SMILES tokenization workflow includes:

  • Build a vocabulary from the SMILES data and assign each token a unique integer. section 10.2.2
  • Protect multi-character elements and groups such as Cl, Br, [nH], and [H] through substitutions before character encoding. section 10.2.2
  • Add start and end markers, map unknown symbols to an unknown token, and encode batches as padded sequences. section 10.2.2

Encoding a molecule

A vocabulary can encode CCO as [<sos>, C, C, O, <eos>] and decode the resulting indices back to CCO. The same representation supports an autoencoder that compresses the sequence into a latent vector and reconstructs the original SMILES. section 10.2.2

Character-level tokenization supplies a simple but weak inductive bias: the model must learn chemical patterns one character at a time. Byte-Pair Encoding and its chemistry-oriented variant, Smiles Pair Encoding, repeatedly merge frequent adjacent tokens, representing recurring motifs or functional groups as single tokens. Shorter, more meaningful sequences can train faster and produce more structured latent spaces with less risk of posterior collapse. section 10.4.4 section 10.5

Figure 10.2 Core components of an autoencoder neural network applied to molecular structures. The process flows from left to right: an input molecule (X) is fed into an encoder network that compresses the molecular representation into a low-dimensional latent vector. This compressed representation captures the most essential features of the molecule in a compact form. The decoder network then attempts to reconstruct the original molecule (X') from this latent vector. The reconstruction loss (shown by the dashed red arrow) measures the difference between the input molecule and the reconstructed output, providing the training signal that forces the network to learn meaningful molecular representations. The information bottleneck created by the latent vector ensures the autoencoder must learn to encode only the most important molecular features necessary for accurate reconstruction.

Use the autoencoder diagram to connect tokenized molecular input with encoding, latent compression, decoding, and reconstruction loss. section 10.2.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains vocabulary construction, substitutions, special tokens, encoding, batching, and the shift from characters to learned molecular subunits.
Examples
Uses CCO, benzene notation, chlorine, bromine, aromatic nitrogen, and explicit hydrogen as tokenization examples.
What it assumes
Assumes SMILES strings as molecular inputs and teaches their conversion into numerical sequences for neural networks.
What you can do after
Build it from scratch by creating a Python/PyTorch vocabulary and encoding, decoding, and batching SMILES sequences.
How it connects
Connects tokenization to SMILES, autoencoders, latent spaces, inductive bias, posterior collapse, and recurrent neural networks.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.