A variational autoencoder (VAE) learns a compressed, continuous representation of data that can be sampled to generate new examples. In molecular design, it can map molecules into a latent space: a simplified set of coordinates for exploring possible chemical structures. section 10.0 section 10.3
A basic autoencoder compresses each molecule to one point and trains a decoder to reconstruct it. The decoder may perform well at those points but produce invalid output from unexplored coordinates between them. A VAE instead encodes each molecule as a probability distribution, such as a Gaussian with a predicted mean and variance. Training samples latent vectors from that distribution, so the decoder learns to reconstruct from a neighborhood rather than a single point. section 10.2.4 section 10.3
VAE training balances two objectives:
- Reconstruction loss measures whether the decoder can rebuild the input molecular SMILES string from a sampled latent vector. SMILES is a text representation of a molecule. section 10.3
- Kullback–Leibler (KL) divergence measures how far the encoded distribution is from a fixed prior, here a standard normal distribution. Pulling distributions toward a common prior helps organize the latent space for sampling. section 10.3 section 10.5
The total loss adds reconstruction loss to KL divergence multiplied by a weight, β. Too much emphasis on reconstruction leaves the space less organized; a training failure called posterior collapse occurs when the model ignores the latent space. The reparameterization trick allows training through random sampling, while cyclical annealing adjusts the KL weight during training to help combat collapse. section 10.3 section 10.5
Because SMILES strings are sequences, gated recurrent units (GRUs) can replace simple feedforward networks to capture dependencies in their order. A bidirectional GRU encoder can use context from both directions. section 10.5
Optimizing molecules for a drug target
In the rentosertib program, the GENTRL algorithm used a VAE to learn a continuous chemical latent space. A reinforcement learning agent explored that space using rewards from property-prediction models. The rewards balanced predicted target binding with properties such as selectivity, solubility, and synthetic accessibility; medicinal chemists adjusted priorities as designs were evaluated. section 13.3.1














