liveBook · Manning

Word dropout

As explained in Build AI Drug Discovery Pipelines

Word dropout is a regularization technique that randomly replaces input tokens with an unknown-token marker during training.

Word dropout regularizes sequence models by randomly replacing input tokens with an unknown-token marker during training. In molecular generation, this makes an encoder rely on surrounding SMILES context instead of memorizing individual tokens. section 10.4.3

Apply word dropout during the training pass as follows:

  • Create a random boolean mask for token positions, selecting each position with the configured word-dropout probability. section 10.4.3
  • Replace selected input indices with the vocabulary's <UNK> index, while leaving the target sequence available for reconstruction. section 10.4.3
  • Pass the corrupted sequence through the encoder, sample the latent vector, and decode with teacher forcing. section 10.4.3
  • Restrict the corruption to training mode; evaluation uses the unmodified input sequence. section 10.4.3
Listing 10.11 The forward Method: The End-to-End Pass
def forward(
    self, x: torch.Tensor, target: torch.Tensor, 
    lengths: Optional[torch.Tensor] = None
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:   #A
    
    if self.training:
        self.step_count += 1   #B
 
    x_dropout = x
    
    if self.training and (
        self.word_dropout > 0
    ):   #C
        
        dropout_mask = (
            torch.rand_like(x.float()) < self.word_dropout
        )   #D
        
        x_dropout = x.masked_fill(
            dropout_mask, self.vocab.unk_idx
        )   #E
        
    mu, logvar = self.encode(
        x_dropout, lengths
    )   #F
    
    z = self.reparameterize(
        mu, logvar
    )   #G
    
    decoder_input = target[:, :-1]
    
    logits = self.decode(
        decoder_input, z
    )   #H
    
    return (
        logits, mu, logvar
    )   #I

Inspect the end-to-end forward pass showing conditional masking, <UNK> replacement, encoding, latent sampling, and teacher-forced decoding. section 10.4.3

VAE-CYC configuration

A VAE-CYC configuration combines word dropout of 0.1 with SmilesPE tokenization, encoder-decoder dropout, cyclical KL annealing, and free bits to support meaningful molecular latent representations. section 10.4.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains masking, <UNK> replacement, training-mode gating, and downstream encoding, but does not develop a standalone theoretical analysis.
Examples
Uses <UNK> token replacement in a VAE-CYC molecular generator and configures word dropout alongside SmilesPE tokenization.
What it assumes
Assumes tokenized sequences, a vocabulary with an unknown-token index, and an encoder-decoder training loop.
What you can do after
Apply word dropout through a PyTorch forward pass that masks tokens only during training.
How it connects
Connects regularization with SMILES tokenization, encoder-decoder models, teacher forcing, variational autoencoders, and latent representations.

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.