Cyclical annealing is a VAE training schedule that repeatedly raises the KL-divergence weight from zero to a maximum, then resets it. The model initially prioritizes reconstructing SMILES, while later cycles strengthen latent-space regularization. Repeating this pressure helps prevent posterior collapse, where the decoder ignores the latent vector, while encouraging a smooth, continuous chemical latent space. section 10.3 section 10.5
Use the schedule within a molecular VAE workflow:
- Encode tokenized SMILES into a probability distribution with mean and variance, then sample a latent vector using the reparameterization trick. section 10.3
- Decode the sampled vector into a reconstructed SMILES sequence and combine reconstruction loss with β-weighted KL divergence. section 10.3
- Change β across repeated cycles so the decoder learns to use latent information before stronger regularization pulls molecular distributions toward the standard-normal prior. section 10.3 section 10.4.5
- Generate molecules by sampling latent vectors from the prior and evaluate validity, uniqueness, novelty, and diversity. section 10.4.5
VAE-CYC configuration
A PyTorch-style molecular VAE can expose max_kl_weight, n_cycles, and cycle_ratio alongside latent dimensions, GRU architecture, word dropout, and free bits. These settings make the annealing policy part of the model configuration rather than an implicit training detail. section 10.4.5
Listing 10.13 Defining the Model Architecture
class VAECYC(
AbstractMolecularAutoencoder
): #A
def __init__(
self, vocab: Vocab, latent_dim: int = 64, max_length: int = 100,
embed_dim: int = 128, hidden_dim: int = 512,
encoder_num_layers: int = 2, encoder_bidirectional: bool = True,
encoder_dropout: float = 0.2, decoder_num_layers: int = 4,
decoder_dropout: float = 0.2, max_kl_weight: float = 0.6,
n_cycles: int = 20, cycle_ratio: float = 0.7,
word_dropout: float = 0.2, free_bits: float = 0.25,
max_epochs: int = 10
): #B
super().__init__()
self.vocab = vocab
self.latent_dim = latent_dim #C
self.embedding = nn.Embedding(
num_embeddings=self.vocab_size,
embedding_dim=self.embed_dim,
padding_idx=vocab.pad_idx #D
)
self.encoder = nn.GRU(
input_size=self.embed_dim,
hidden_size=self.hidden_dim,
num_layers=self.encoder_num_layers,
batch_first=True, #E
bidirectional=self.encoder_bidirectional, #F
dropout= \
self.encoder_dropout if self.encoder_num_layers > 1 else 0
)
encoder_output_dim = self.hidden_dim * (
2 if self.encoder_bidirectional else 1
) #G
self.fc_mu = nn.Linear(
encoder_output_dim, self.latent_dim
) #H
self.fc_logvar = nn.Linear(
encoder_output_dim, self.latent_dim
) #I
self.decoder = nn.GRU(
input_size=self.embed_dim + self.latent_dim, #J
hidden_size=self.hidden_dim,
num_layers=self.decoder_num_layers,
batch_first=True,
bidirectional=False, #K
dropout= \
self.decoder_dropout if self.decoder_num_layers > 1 else 0
)
self.latent_to_hidden = nn.Linear(
self.latent_dim,
self.hidden_dim * self.decoder_num_layers #L
)
self.output_projection = nn.Linear(
self.hidden_dim, self.vocab_size
) #M
self.register_buffer(
'step_count', torch.tensor(0, dtype=torch.long)
) #N
self._initialize_weights() #OInspect the model constructor to see where the maximum KL weight, number of cycles, and cycle ratio are configured alongside the encoder and decoder. section 10.4.5
