liveBook · Manning

ReLU

As explained in Build AI Drug Discovery Pipelines

ReLU is an activation function that outputs zero for negative inputs and otherwise returns the input value.

The Rectified Linear Unit (ReLU) is an activation function: it returns zero for a negative input and otherwise returns the input unchanged. For example, ReLU(-2) = 0 and ReLU(3) = 3. Its nonlinearity helps a neural network learn complex relationships rather than simply stacking linear transformations. section 8.2.2 section 11.5.2

In drug discovery, a neural network can use ReLU between fully connected layers to learn relationships between molecular features and binding affinity. An active-learning surrogate model, for example, takes numerical molecular fingerprints as input and predicts docking scores, providing a proxy for a more expensive process. section 9.2.2

Listing 9.5 Deep Learning Surrogate Model for Binding Affinity Prediction
class DeepDockingModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size=1):
        super(DeepDockingModel, self).__init__()
        
        self.layers = nn.Sequential(    #A
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(hidden_size, hidden_size // 2),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(hidden_size // 2, output_size)
        )
 
    def forward(self, x):
        return self.layers(x)
 
    def predict(self, fingerprints, device):
        self.eval()    #B
        with torch.no_grad():    #C
            inputs = torch.tensor(
                np.atleast_2d(fingerprints), dtype=torch.float32
            ).to(device)    #D
            outputs = self(inputs)    #E
            return outputs.cpu().numpy().flatten()

The PyTorch surrogate model places nn.ReLU() after each hidden linear layer, followed by dropout. Its final linear layer produces the prediction without a ReLU. section 9.2.2

ReLU is also used after graph convolution layers, which combine information from local neighborhoods in molecular graphs. The resulting nonlinear features can help a model learn from graph structure. section 11.5.2

Listing 11.7 Process Molecular Graph through GCN Layers
def forward_mol(self, data_mol):
    mol_x, mol_edge_index, mol_batch = \
        data_mol.x, data_mol.edge_index, data_mol.batch
 
    x = self.relu(self.mol_conv1(mol_x, mol_edge_index))
    x = self.relu(self.mol_conv2(x, mol_edge_index))
    x = self.relu(self.mol_conv3(x, mol_edge_index))
 
    x = global_mean_pool(x, mol_batch)
 
    x = self.relu(self.mol_fc_g1(x))
    x = self.dropout(x)
    x = self.mol_fc_g2(x)
    x = self.dropout(x)
    return x

The molecular graph branch applies ReLU after three graph convolution layers and again after a fully connected layer. section 11.5.2

Activation choice affects training because backpropagation multiplies derivatives through the network. Sigmoid and tanh can have near-zero gradients at their flattened extremes, contributing to vanishing gradients. ReLU is chosen to help keep gradient updates stable, but initialization still matters: Kaiming (He) initialization accounts for negative values being zeroed and aims to maintain activation variance across layers. section 8.2.2 section 10.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains activation behavior and gradient motivation, then demonstrates placement in network forward passes.
Examples
Uses a kinase binder network and a molecular-fingerprint surrogate predicting docking scores.
What you can do after
Apply it via PyTorch using nn.ReLU() between neural network layers.
How it connects
Connects activation functions to backpropagation, vanishing gradients, graph convolution, and Kaiming initialization.

Also explained in

Compare how 16 books explain ReLU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.