Word dropout regularizes sequence models by randomly replacing input tokens with an unknown-token marker during training. In molecular generation, this makes an encoder rely on surrounding SMILES context instead of memorizing individual tokens. section 10.4.3
Apply word dropout during the training pass as follows:
- Create a random boolean mask for token positions, selecting each position with the configured word-dropout probability. section 10.4.3
- Replace selected input indices with the vocabulary's <UNK> index, while leaving the target sequence available for reconstruction. section 10.4.3
- Pass the corrupted sequence through the encoder, sample the latent vector, and decode with teacher forcing. section 10.4.3
- Restrict the corruption to training mode; evaluation uses the unmodified input sequence. section 10.4.3
Listing 10.11 The forward Method: The End-to-End Pass
def forward(
self, x: torch.Tensor, target: torch.Tensor,
lengths: Optional[torch.Tensor] = None
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: #A
if self.training:
self.step_count += 1 #B
x_dropout = x
if self.training and (
self.word_dropout > 0
): #C
dropout_mask = (
torch.rand_like(x.float()) < self.word_dropout
) #D
x_dropout = x.masked_fill(
dropout_mask, self.vocab.unk_idx
) #E
mu, logvar = self.encode(
x_dropout, lengths
) #F
z = self.reparameterize(
mu, logvar
) #G
decoder_input = target[:, :-1]
logits = self.decode(
decoder_input, z
) #H
return (
logits, mu, logvar
) #IInspect the end-to-end forward pass showing conditional masking, <UNK> replacement, encoding, latent sampling, and teacher-forced decoding. section 10.4.3
VAE-CYC configuration
A VAE-CYC configuration combines word dropout of 0.1 with SmilesPE tokenization, encoder-decoder dropout, cyclical KL annealing, and free bits to support meaningful molecular latent representations. section 10.4.5
