liveBook · Manning

Decoder

As explained in Build AI Drug Discovery Pipelines

A decoder is the component of a sequence model that generates an output sequence from an internal representation and previously processed tokens.

A decoder turns an internal representation into an output. In a molecular autoencoder, it takes a compressed latent vector and tries to reconstruct the original molecule’s SMILES string, a text representation of its structure. Training minimizes the difference between the input and reconstruction. section 10.2.2

Decoders can produce outputs in different ways. A basic molecular autoencoder expands the latent vector through neural-network layers and outputs scores for each token position. A sequence-to-sequence decoder instead generates a SMILES string one token at a time, using its evolving hidden state to carry information forward. section 10.2.3 section 10.4.1

Listing 10.4 Basic Autoencoder Architecture: Decoder
        decoder_layers = []
        input_dim = latent_dim    #A
        
        for hidden_dim in reversed(hidden_dims):    #B
            decoder_layers.extend([
                nn.Linear(input_dim, hidden_dim),
                nn.ReLU(),
                nn.BatchNorm1d(hidden_dim),
                nn.Dropout(0.1)
            ])
            input_dim = hidden_dim
        
        decoder_layers.extend([    #C
            nn.Linear(input_dim, vocab_size * max_length),
            nn.Unflatten(1, (max_length, vocab_size))    #D
        ])
        self.decoder = nn.Sequential(*decoder_layers)
        self._initialize_weights()    #E

Inspect a PyTorch decoder that expands a latent vector into token scores for a fixed maximum sequence length. section 10.2.3

In a molecular variational autoencoder (VAE), a bidirectional GRU encoder produces parameters for a latent distribution, from which a vector is sampled. A unidirectional GRU decoder uses that vector to generate the output SMILES. Its initial hidden state comes from the vector, which is also joined to token embeddings at each step. section 10.4.3

Listing 10.10 Unidirectional Decoder with Latent Concatenation
def decode(
    self, x: torch.Tensor, z: torch.Tensor
) -> torch.Tensor:    #A
    batch_size, seq_len = x.size()
    
    h_init = self.latent_to_hidden(z)    #B
    h = h_init.view(batch_size, self.decoder_num_layers, self.hidden_dim)
    h = h.transpose(0, 1).contiguous()    #C
    
    embedded = self.embedding(x)    #D
    
    z_expanded = z.unsqueeze(1).expand(
        batch_size, seq_len, self.latent_dim
    )    #E
    
    decoder_input = torch.cat(
        [embedded, z_expanded], dim=2
    )    #F
    output, _ = self.decoder(decoder_input, h)    #G
    logits = self.output_projection(output)    #H
    return logits           

Follow the decoder’s latent-to-hidden projection, token embedding, latent concatenation, GRU, and output projection. section 10.4.3

During training, teacher forcing supplies ground-truth tokens to the recurrent decoder. During generation, it starts with a start token and feeds its predictions back as inputs until it produces an end token. The basic autoencoder’s reconstruction loss uses cross-entropy and ignores padding tokens. section 10.4.3 section 10.4.1 section 10.2.3

Reconstruction alone is not enough for molecular design. A decoded SMILES must also be chemically valid: RDKit should be able to parse it into a molecule. Invalid outputs cannot provide meaningful molecular-property feedback to a reinforcement-learning agent. Structural similarity can help assess how faithfully an encoded molecule is reconstructed. section 10.2.1

A standard Transformer also has a decoder, but it uses cross-attention to consult the encoder’s token representations while generating a target sequence. An encoder-only Transformer omits the decoder when the goal is to represent an input, rather than generate a new sequence. section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Develops feed-forward and recurrent decoder implementations, while contrasting them briefly with a Transformer decoder.
Examples
Implements a fixed-length SMILES reconstruction decoder and a latent-conditioned GRU decoder.
What it assumes
Introduces SMILES tokenization, latent vectors, and recurrent sequence processing before applying them to molecular decoding.
What you can do after
Build a molecular decoder from scratch using PyTorch layers and assess its reconstruction and molecular validity.
How it connects
Connects decoders to autoencoders, latent space, GRUs, teacher forcing, and Transformer cross-attention.

Also explained in

Compare how 9 books explain decoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.