liveBook · Manning

Activation function

As explained in Grokking Machine Learning, Second Edition

An activation function is a function applied to a neural network unit's input to produce its output and introduce a nonlinear transformation.

An activation function takes a unit’s weighted-input score and turns it into that unit’s output. In a perceptron, inputs are multiplied by weights, added together with a bias, and then passed through the activation function to produce a prediction. This small final step is what lets a model turn a score into a useful output. section 10.5

Start with the step function: negative scores become 0, while scores of 0 or more become 1. It provides a compact way to make a yes-or-no prediction. The step function is an activation function, but neural networks trained with gradient descent do not use it in practice because it prevents gradient descent from working. section 5.8 section 10.6

Figure 10.19 Three different activation functions used in neural networks. Left: The sigmoid function, represented by the Greek letter sigma. Middle: The hyperbolic tangent, or tanh. Right: The rectified linear unit, or ReLU.

Compare sigmoid, tanh, and ReLU by looking at how each reshapes an input number. section 10.14

Common choices handle outputs in different ways:

  • Sigmoid maps any real-valued input to a value between 0 and 1. It can make a classification output interpretable, but its flat ends can produce very small derivatives. section 10.25
  • Tanh maps inputs to the interval from -1 to 1. It is less flat than sigmoid, yet can still suffer from vanishing gradients. section 10.14
  • ReLU returns 0 for a negative input and returns the input itself when it is nonnegative. Its derivative is 1 for positive inputs, helping reduce the vanishing-gradient problem. section 10.14
  • Softmax can be used at a classification network’s output layer, as in a two-class model with two output nodes. section 10.18

Activation functions matter most when perceptrons are joined into layers. Used between layers, they break linearity, allowing a neural network to form more complicated decision boundaries than a stack of purely linear operations. A common setup uses ReLU in hidden nodes and sigmoid at the final node for classification. section 10.25 section 10.14

Classifying points in a circle

For a circular two-class dataset, use two hidden layers with 128 and 64 nodes, each activated by ReLU, followed by a two-node softmax output layer. In Keras, the activation is set directly on each Dense layer, such as activation='relu' or activation='softmax'. The resulting model reaches 0.9000 training accuracy after 100 epochs in the shown run. section 10.18

The final activation should match the task. A sigmoid output restricts predictions to 0 through 1 for classification. For regression, remove the final sigmoid so the network can return any numerical value; the remaining training process can stay the same. section 10.20

Choosing activation functions is also part of choosing a neural-network architecture. Along with layer and node counts, they are hyperparameters that can be tuned to improve training. section 10.16

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines step, sigmoid, tanh, and ReLU, explains gradient effects, and implements layer choices in Keras.
Examples
Uses a step-function classifier, a weighted perceptron, a circular dataset, and Hyderabad housing-price regression.
What it assumes
Introduces activation functions through perceptron scores before extending them to neural-network layers.
What you can do after
Apply it via Keras by selecting ReLU and softmax layers for classification or omitting the final activation for regression.
How it connects
Connects perceptrons, neural networks, gradient descent, backpropagation, vanishing gradients, classification, and regression.

Also explained in

Compare how 20 books explain activation function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.