liveBook · Manning

Vocabulary

As explained in Build AI Drug Discovery Pipelines

A vocabulary is a mapping between the permitted text tokens and their integer identifiers.

A vocabulary maps permitted SMILES tokens to unique integer identifiers so neural networks can process molecular strings numerically. Character-level vocabularies scan training SMILES, collect unique characters, and define both token-to-index and index-to-token mappings. section 10.2.2

A practical molecular vocabulary handles several representation problems:

  • Substitutes multi-character units such as Cl, Br, [nH], and [H] with single reserved characters before encoding. section 10.2.2
  • Adds <sos> and <eos> markers to identify sequence boundaries, <pad> for fixed-length batches, and <unk> for unseen characters. section 10.2.2
  • Encodes SMILES into integer sequences and decodes those sequences back into strings. section 10.2.2

Encoding a molecule

A vocabulary can encode the SMILES string CCO as [<sos>, C, C, O, <eos>] and represent it with integer indices such as [1, 4, 4, 6, 2]. Decoding the indices reconstructs CCO. section 10.2.2

For larger molecular models, a vocabulary can use Smiles Pair Encoding rather than individual characters. Starting from atom-level tokens, it repeatedly merges frequent adjacent pairs, producing longer tokens that capture recurring chemical motifs and shorten sequences. section 10.4.4

Listing 10.1 The Vocabulary Class
import torch
import torch.nn as nn
from typing import List, Union, Optional, Tuple
from collections import Counter
 
class Vocab:
    """Vocabulary class for SMILES tokenization in molecular autoencoders.
    
    Example:
        >>> vocab = Vocab(['CCO', 'c1ccccc1'])
        >>> tokens = vocab.encode('CCO')
        >>> print(tokens)  # [1, 4, 4, 6, 2]  # <sos>, C, C, O, <eos>
        >>> smiles = vocab.decode(tokens)
        >>> print(smiles)  # 'CCO'
    """
    
    def __init__(self, smiles_list: Optional[List[str]] = None):
        """Initialize vocabulary from SMILES data."""
        self.char_map = {    #A
            'Cl': 'Q',   # Chlorine
            'Br': 'W',   # Bromine  
            '[nH]': 'X', # Pyrrole nitrogen
            '[H]': 'Y'   # Explicit hydrogen
        }
        
        self.special_tokens = [    #B
            '<pad>',  # Padding token for batch processing
            '<sos>',  # Start of sequence
            '<eos>',  # End of sequence  
            '<unk>'   # Unknown character (out-of-vocabulary)
        ]
        
        self.char_to_idx = {}    #C
        self.idx_to_char = {}
        self.vocab_size = 0
        
        if smiles_list:
            self.build_vocab(smiles_list)    #D

Inspect the vocabulary implementation, including substitutions, special-token indices, and the encode-decode interface. section 10.2.2

Listing 10.2 Encoding SMILES
    def encode(self, smiles, add_special_tokens=True):
        """Convert SMILES string to sequence of token indices."""        
        preprocessed = self._apply_substitutions(smiles, self.char_map)
        tokens = []
        
        if add_special_tokens:
            tokens.append(self.sos_idx)  # Start token
        
        for char in preprocessed:    #A
            tokens.append(self.char_to_idx.get(char, self.unk_idx))
        
        if add_special_tokens:
            tokens.append(self.eos_idx)  # End token
            
        return tokens
 
    def encode_batch(self, smiles_list, max_length=None):
        """Encode multiple SMILES strings into a padded tensor."""        
        encoded_sequences = [self.encode(smiles) for smiles in smiles_list]
        
        if max_length is None:    #B
            max_length = max(
                (len(seq) for seq in encoded_sequences), default=0)
        
        batch_size = len(encoded_sequences)
        padded_tensor = torch.full(
            (batch_size, max_length), self.pad_idx, dtype=torch.long
        )    #C
        
        for i, sequence in enumerate(encoded_sequences):
            seq_len = min(len(sequence), max_length)
            padded_tensor[i, :seq_len] = torch.tensor(sequence[:seq_len])
        
        return padded_tensor

Follow the encoding workflow from SMILES preprocessing through special-token insertion and integer lookup. section 10.2.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains character mappings, substitutions, special tokens, encoding, decoding, padding, and BPE-style vocabulary construction.
Examples
Uses CCO, aromatic benzene SMILES, halogen substitutions, special tokens, and frequent-pair merges as concrete examples.
What it assumes
Assumes SMILES strings and neural-network inputs, while defining tokenization, integer identifiers, special tokens, and sequence padding.
What you can do after
Build it from scratch: encode, pad, batch, and decode SMILES sequences with character-level vocabulary logic.
How it connects
Connects tokenization with SMILES, molecular autoencoders, latent representations, padding, and Byte-Pair Encoding.

Also explained in

Compare how 17 books explain vocabulary →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.