The Rectified Linear Unit (ReLU) is an activation function: it returns zero for a negative input and otherwise returns the input unchanged. For example, ReLU(-2) = 0 and ReLU(3) = 3. Its nonlinearity helps a neural network learn complex relationships rather than simply stacking linear transformations. section 8.2.2 section 11.5.2
In drug discovery, a neural network can use ReLU between fully connected layers to learn relationships between molecular features and binding affinity. An active-learning surrogate model, for example, takes numerical molecular fingerprints as input and predicts docking scores, providing a proxy for a more expensive process. section 9.2.2
Listing 9.5 Deep Learning Surrogate Model for Binding Affinity Prediction
class DeepDockingModel(nn.Module):
def __init__(self, input_size, hidden_size, output_size=1):
super(DeepDockingModel, self).__init__()
self.layers = nn.Sequential( #A
nn.Linear(input_size, hidden_size),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(hidden_size, hidden_size // 2),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(hidden_size // 2, output_size)
)
def forward(self, x):
return self.layers(x)
def predict(self, fingerprints, device):
self.eval() #B
with torch.no_grad(): #C
inputs = torch.tensor(
np.atleast_2d(fingerprints), dtype=torch.float32
).to(device) #D
outputs = self(inputs) #E
return outputs.cpu().numpy().flatten()The PyTorch surrogate model places nn.ReLU() after each hidden linear layer, followed by dropout. Its final linear layer produces the prediction without a ReLU. section 9.2.2
ReLU is also used after graph convolution layers, which combine information from local neighborhoods in molecular graphs. The resulting nonlinear features can help a model learn from graph structure. section 11.5.2
Listing 11.7 Process Molecular Graph through GCN Layers
def forward_mol(self, data_mol):
mol_x, mol_edge_index, mol_batch = \
data_mol.x, data_mol.edge_index, data_mol.batch
x = self.relu(self.mol_conv1(mol_x, mol_edge_index))
x = self.relu(self.mol_conv2(x, mol_edge_index))
x = self.relu(self.mol_conv3(x, mol_edge_index))
x = global_mean_pool(x, mol_batch)
x = self.relu(self.mol_fc_g1(x))
x = self.dropout(x)
x = self.mol_fc_g2(x)
x = self.dropout(x)
return xThe molecular graph branch applies ReLU after three graph convolution layers and again after a fully connected layer. section 11.5.2
Activation choice affects training because backpropagation multiplies derivatives through the network. Sigmoid and tanh can have near-zero gradients at their flattened extremes, contributing to vanishing gradients. ReLU is chosen to help keep gradient updates stable, but initialization still matters: Kaiming (He) initialization accounts for negative values being zeroed and aims to maintain activation variance across layers. section 8.2.2 section 10.2.3















