An activation function transforms a neuron's weighted input into its output. In a neural network, nonlinear activations in hidden layers help the model capture nonlinear relationships. section 8.2.1
A single logistic neuron uses the sigmoid activation to produce a probability, but its decision boundary is still linear in the original feature space. Stacking neurons with nonlinear activations allows a network to represent more complex relationships. section 8.2.1 section 8.2.1
Activation choice also affects how a network learns:
- Sigmoid maps inputs to (0, 1). At very large or small inputs, its curve flattens and its gradients approach zero. section 8.2.2
- Hyperbolic tangent (tanh) maps inputs to (-1, 1). It also saturates and can contribute to vanishing gradients. section 8.2.2
- ReLU is used in the kinase binder network to help keep gradient updates stable during training. section 8.2.2
During backpropagation, derivatives are multiplied through the network. If these terms are repeatedly small, gradients can shrink toward zero, slowing or preventing learning in earlier layers. This is the vanishing gradients problem. section 8.2.2
Figure 8.11 Comparison of activation functions: sigmoid (i.e., logistic), hyperbolic tangent, rectified linear unit (ReLU), and step function. In the earlier computational graphs of a logistic classifier, we used the sigmoid or logistic activation function. In our modeling approach, we switched to using the ReLU activation function.

Compare the shapes of sigmoid, tanh, ReLU, and the step function. section 8.2.2
Listing 8.6 Kinase Binder Neural Network Architecture
class KinaseBinderNN(nn.Module):
def __init__(self, input_size=2048):
super(KinaseBinderNN, self).__init__()
self.fc1 = nn.Linear(input_size, 512)
self.fc2 = nn.Linear(512, 256)
self.fc3 = nn.Linear(256, 1)
self.relu = nn.ReLU()
self.dropout = nn.Dropout(0.2)
def forward(self, x):
x = self.dropout(self.relu(self.fc1(x)))
x = self.dropout(self.relu(self.fc2(x)))
x = self.fc3(x)
return x
In this PyTorch kinase binder network, ReLU follows each of the first two linear layers; the final linear layer returns the output. section 8.2.2



















