liveBook · Manning

Feedforward network

As explained in Build AI Drug Discovery Pipelines

A position-wise feed-forward network is a neural network applied independently to each sequence position, commonly using two linear transformations and a nonlinear activation.

A feedforward network passes information from one layer to the next, without feeding it back during prediction. Each layer transforms its input; a fully connected, or dense, layer connects every neuron to every neuron in the preceding layer. section 8.2.1

In a Transformer encoder, a position-wise feedforward network (FFN) applies the same small network independently to each sequence position. It typically uses two linear transformations with a nonlinear activation, such as ReLU or GELU, between them. section 12.3.2

For a protein sequence, multi-head self-attention first lets each residue gather context from other positions. The FFN then processes the information at each position. Both sublayers use residual connections and layer normalization, and stacking encoder blocks builds increasingly context-aware representations. section 12.3.2

Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

The encoder stack shows where the position-wise FFN fits in a Transformer used for protein sequences. section 12.3.2

Predicting docking scores

A feedforward surrogate model can predict docking scores from molecular fingerprints, which encode molecular features. A PyTorch implementation uses three fully connected layers, ReLU activations, and dropout at probability 0.5 to help prevent overfitting. This is a molecular-level network, rather than the position-wise FFN in a Transformer. section 9.2.2

Listing 9.5 Deep Learning Surrogate Model for Binding Affinity Prediction
class DeepDockingModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size=1):
        super(DeepDockingModel, self).__init__()
        
        self.layers = nn.Sequential(    #A
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(hidden_size, hidden_size // 2),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(hidden_size // 2, output_size)
        )
 
    def forward(self, x):
        return self.layers(x)
 
    def predict(self, fingerprints, device):
        self.eval()    #B
        with torch.no_grad():    #C
            inputs = torch.tensor(
                np.atleast_2d(fingerprints), dtype=torch.float32
            ).to(device)    #D
            outputs = self(inputs)    #E
            return outputs.cpu().numpy().flatten()

The PyTorch listing assembles the surrogate model with nn.Sequential and passes fingerprints through its layers. section 9.2.2

A plain feedforward network over a flattened SMILES string has no built-in notion of sequence order and handles variable-length molecules awkwardly. This differs from a Transformer encoder, where attention provides sequence context before the position-wise FFN processes it. section 10.4 section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the position-wise FFN as two linear transformations around an activation, and implements a separate three-layer molecular surrogate.
Examples
Uses protein residue representations, molecular fingerprints for docking-score prediction, and flattened SMILES strings to illustrate feedforward networks.
What you can do after
Evaluate and understand the FFN’s role in a protein encoder and apply a feedforward surrogate using PyTorch.
How it connects
Connects feedforward networks to self-attention, residual connections, layer normalization, active learning, and recurrent neural networks.

Also explained in

Compare how 20 books explain feedforward network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.