liveBook · Manning

Activation function

As explained in Build AI Drug Discovery Pipelines

An activation function is a function applied to a neuron's weighted input to produce its output, typically introducing nonlinearity.

An activation function transforms a neuron's weighted input into its output. In a neural network, nonlinear activations in hidden layers help the model capture nonlinear relationships. section 8.2.1

A single logistic neuron uses the sigmoid activation to produce a probability, but its decision boundary is still linear in the original feature space. Stacking neurons with nonlinear activations allows a network to represent more complex relationships. section 8.2.1 section 8.2.1

Activation choice also affects how a network learns:

  • Sigmoid maps inputs to (0, 1). At very large or small inputs, its curve flattens and its gradients approach zero. section 8.2.2
  • Hyperbolic tangent (tanh) maps inputs to (-1, 1). It also saturates and can contribute to vanishing gradients. section 8.2.2
  • ReLU is used in the kinase binder network to help keep gradient updates stable during training. section 8.2.2

During backpropagation, derivatives are multiplied through the network. If these terms are repeatedly small, gradients can shrink toward zero, slowing or preventing learning in earlier layers. This is the vanishing gradients problem. section 8.2.2

Figure 8.11 Comparison of activation functions: sigmoid (i.e., logistic), hyperbolic tangent, rectified linear unit (ReLU), and step function. In the earlier computational graphs of a logistic classifier, we used the sigmoid or logistic activation function. In our modeling approach, we switched to using the ReLU activation function.

Compare the shapes of sigmoid, tanh, ReLU, and the step function. section 8.2.2

Listing 8.6 Kinase Binder Neural Network Architecture
class KinaseBinderNN(nn.Module):
    def __init__(self, input_size=2048):
        super(KinaseBinderNN, self).__init__()
        self.fc1 = nn.Linear(input_size, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 1)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.2)
        
    def forward(self, x):
        x = self.dropout(self.relu(self.fc1(x)))
        x = self.dropout(self.relu(self.fc2(x)))
        x = self.fc3(x)
        return x
 

In this PyTorch kinase binder network, ReLU follows each of the first two linear layers; the final linear layer returns the output. section 8.2.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains neuron transformations, decision boundaries, activation curves, and vanishing gradients, then implements ReLU in a neural network.
Examples
Compares sigmoid, tanh, ReLU, and step curves and uses ReLU in a kinase binder network.
What you can do after
Apply activation functions via PyTorch when constructing a neural network and evaluate their effects on gradient flow.
How it connects
Connects activation functions to logistic regression, computational graphs, hidden layers, backpropagation, and vanishing gradients.

Also explained in

Compare how 20 books explain activation function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.