A latent space is a learned representation space in which data is encoded as coordinates called latent vectors. For molecular design, it can serve as a compressed, continuous map: instead of editing atoms and bonds directly, a model can navigate vectors and decode them into candidate molecules. A point is a representation of a molecule, not the molecule itself. section 10.1 section 10.1.2
An autoencoder learns this map with two networks. The encoder compresses an input into a latent vector; the decoder tries to reconstruct the input from that vector. Training minimizes reconstruction error. The limited size of the vector creates an information bottleneck, encouraging the encoder to retain features useful for reconstruction. section 10.2.2
Encoding a SMILES string
SMILES is a text representation of a molecule. For example, the string CCO can be tokenized into characters, mapped to integer IDs, and converted into learned embedding vectors. An encoder compresses the resulting sequence; a decoder attempts to reproduce it. Token embeddings are inputs to the encoder, while the latent vector represents the compressed sequence. section 10.2.2 section 10.2.3
Figure 10.2 Core components of an autoencoder neural network applied to molecular structures. The process flows from left to right: an input molecule (X) is fed into an encoder network that compresses the molecular representation into a low-dimensional latent vector. This compressed representation captures the most essential features of the molecule in a compact form. The decoder network then attempts to reconstruct the original molecule (X') from this latent vector. The reconstruction loss (shown by the dashed red arrow) measures the difference between the input molecule and the reconstructed output, providing the training signal that forces the network to learn meaningful molecular representations. The information bottleneck created by the latent vector ensures the autoencoder must learn to encode only the most important molecular features necessary for accurate reconstruction.

Trace a molecule through the encoder, latent bottleneck, decoder, and reconstruction loss. section 10.2.2
A useful molecular latent space should support three related goals:
- Validity: decoded vectors should yield chemically valid molecules, such as SMILES strings that RDKit can parse. section 10.2.1
- Reconstruction: encoding and decoding an input should preserve its structure; structural similarity can help measure this. section 10.2.1
- Continuity: small changes in a latent vector should correspond to small changes in molecular structure, making navigation more useful. section 10.2.4
Good reconstruction alone does not guarantee useful generation. A basic autoencoder can reconstruct familiar molecules accurately while leaving an unstructured space of isolated regions; sampling between them can produce invalid molecules. High token accuracy is also insufficient: a single incorrect SMILES token can make a structure invalid. section 10.5 section 10.2.4
A variational autoencoder (VAE) addresses this by encoding each molecule as a probability distribution rather than a single point. It samples a vector from that distribution and trains the decoder to reconstruct the molecule. A KL-divergence term pulls the distributions toward a shared prior, encouraging a more continuous space. Balancing this term with reconstruction matters: the decoder can otherwise learn to ignore the latent vector, a failure called posterior collapse. section 10.3 section 10.5
In a molecular VAE-CYC pipeline, SMILES tokens pass through a bidirectional GRU encoder, which produces a latent distribution. A sampled vector conditions a unidirectional GRU decoder that generates tokens one at a time. Cyclical annealing adjusts the KL weight during training to help prevent posterior collapse. New candidates can be generated by sampling vectors from the prior and decoding them. section 10.4.3 section 10.4.5
Evaluate the map from several angles: reconstruction quality, continuity, and the validity and diversity of generated molecules. For targeted design, a reinforcement learning agent can propose moves in latent space and receive rewards based on decoded molecules’ predicted properties. The reward must be chosen carefully: an agent may exploit a mismatch between the score and the intended design goal. section 10.2.4 section 10.1.3
Figure 10.1 Overview of targeted molecular generation with latent reinforcement learning. The system begins with an input molecule that is encoded into a compressed latent vector representation. An RL agent operates within this latent space, proposing optimization actions to navigate toward regions associated with desired molecular properties (such as "Active" and "Non-toxic" regions shown). The decoder then reconstructs these optimized latent vectors back into novel molecular structures. Rewards are calculated based on the properties of the generated molecules and fed back to improve the RL agent's decision-making. This approach enables continuous optimization in latent space rather than discrete structural modifications, allowing guided generation of molecules with specific target properties.

Follow an encoded molecule through latent-space optimization, decoding, and reward feedback. section 10.1.3





















