A vocabulary maps permitted SMILES tokens to unique integer identifiers so neural networks can process molecular strings numerically. Character-level vocabularies scan training SMILES, collect unique characters, and define both token-to-index and index-to-token mappings. section 10.2.2
A practical molecular vocabulary handles several representation problems:
- Substitutes multi-character units such as Cl, Br, [nH], and [H] with single reserved characters before encoding. section 10.2.2
- Adds <sos> and <eos> markers to identify sequence boundaries, <pad> for fixed-length batches, and <unk> for unseen characters. section 10.2.2
- Encodes SMILES into integer sequences and decodes those sequences back into strings. section 10.2.2
Encoding a molecule
A vocabulary can encode the SMILES string CCO as [<sos>, C, C, O, <eos>] and represent it with integer indices such as [1, 4, 4, 6, 2]. Decoding the indices reconstructs CCO. section 10.2.2
For larger molecular models, a vocabulary can use Smiles Pair Encoding rather than individual characters. Starting from atom-level tokens, it repeatedly merges frequent adjacent pairs, producing longer tokens that capture recurring chemical motifs and shorten sequences. section 10.4.4
Listing 10.1 The Vocabulary Class
import torch
import torch.nn as nn
from typing import List, Union, Optional, Tuple
from collections import Counter
class Vocab:
"""Vocabulary class for SMILES tokenization in molecular autoencoders.
Example:
>>> vocab = Vocab(['CCO', 'c1ccccc1'])
>>> tokens = vocab.encode('CCO')
>>> print(tokens) # [1, 4, 4, 6, 2] # <sos>, C, C, O, <eos>
>>> smiles = vocab.decode(tokens)
>>> print(smiles) # 'CCO'
"""
def __init__(self, smiles_list: Optional[List[str]] = None):
"""Initialize vocabulary from SMILES data."""
self.char_map = { #A
'Cl': 'Q', # Chlorine
'Br': 'W', # Bromine
'[nH]': 'X', # Pyrrole nitrogen
'[H]': 'Y' # Explicit hydrogen
}
self.special_tokens = [ #B
'<pad>', # Padding token for batch processing
'<sos>', # Start of sequence
'<eos>', # End of sequence
'<unk>' # Unknown character (out-of-vocabulary)
]
self.char_to_idx = {} #C
self.idx_to_char = {}
self.vocab_size = 0
if smiles_list:
self.build_vocab(smiles_list) #DInspect the vocabulary implementation, including substitutions, special-token indices, and the encode-decode interface. section 10.2.2
Listing 10.2 Encoding SMILES
def encode(self, smiles, add_special_tokens=True):
"""Convert SMILES string to sequence of token indices."""
preprocessed = self._apply_substitutions(smiles, self.char_map)
tokens = []
if add_special_tokens:
tokens.append(self.sos_idx) # Start token
for char in preprocessed: #A
tokens.append(self.char_to_idx.get(char, self.unk_idx))
if add_special_tokens:
tokens.append(self.eos_idx) # End token
return tokens
def encode_batch(self, smiles_list, max_length=None):
"""Encode multiple SMILES strings into a padded tensor."""
encoded_sequences = [self.encode(smiles) for smiles in smiles_list]
if max_length is None: #B
max_length = max(
(len(seq) for seq in encoded_sequences), default=0)
batch_size = len(encoded_sequences)
padded_tensor = torch.full(
(batch_size, max_length), self.pad_idx, dtype=torch.long
) #C
for i, sequence in enumerate(encoded_sequences):
seq_len = min(len(sequence), max_length)
padded_tensor[i, :seq_len] = torch.tensor(sequence[:seq_len])
return padded_tensorFollow the encoding workflow from SMILES preprocessing through special-token insertion and integer lookup. section 10.2.2
















