Teacher forcing trains a sequence model using the correct preceding tokens as inputs instead of its own predictions. For molecular generation, it helps stabilize training as a model learns to reconstruct SMILES strings, a text representation of molecules. section 10.2.3
To prepare a training pair, tokenize a SMILES sequence and convert its tokens to integer IDs. Use all but the last token as input and all but the first as the target. Each input position is paired with the next correct token. section 10.2.3
Shifted SMILES tokens
For the token sequence <sos>, C, C, O, <eos>, the input is <sos>, C, C, O and the target is C, C, O, <eos>. The model receives the correct preceding token at each position and learns to predict the next one. section 10.2.3
In a sequence-to-sequence variational autoencoder, the decoder embeds the supplied tokens and combines each embedding with the molecule’s latent vector, a compressed representation. A GRU then processes these inputs and produces token predictions. During generation, the decoder instead proceeds from generated tokens, creating a difference between training and generation. section 10.4.3
Listing 10.11 The forward Method: The End-to-End Pass
def forward(
self, x: torch.Tensor, target: torch.Tensor,
lengths: Optional[torch.Tensor] = None
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: #A
if self.training:
self.step_count += 1 #B
x_dropout = x
if self.training and (
self.word_dropout > 0
): #C
dropout_mask = (
torch.rand_like(x.float()) < self.word_dropout
) #D
x_dropout = x.masked_fill(
dropout_mask, self.vocab.unk_idx
) #E
mu, logvar = self.encode(
x_dropout, lengths
) #F
z = self.reparameterize(
mu, logvar
) #G
decoder_input = target[:, :-1]
logits = self.decode(
decoder_input, z
) #H
return (
logits, mu, logvar
) #IThe forward pass selects all but the final target token as decoder input before calling decode. section 10.4.3


