liveBook · Manning

Activation function

As explained in Machines that Think

An activation function is a mathematical function that transforms a computational unit's combined input signal, typically nonlinearly, into its response.

An activation function transforms a neuron's combined input into its response. By introducing nonlinearity, it helps a neural network model patterns and boundaries that weighted sums alone cannot capture. section 3.1.1 section 3.6

A neuron scales its inputs by weights, adds the results and a bias, then applies an activation function. The resulting output can pass information to other parts of the network. The function shapes that response: it can pass through, reduce, or suppress a signal. section 3.1.1

Figure 3.5 A deep network without activation functions collapses into a single linear transformation. No matter how many layers are stacked, their combined effect is equivalent to a single layer, which would limit the model to learning only simple, linearly separable patterns.

Without nonlinear activation functions, stacking layers still produces only a single linear transformation. section 3.1.4

Common activation functions shape signals in different ways:

  • Sigmoid maps inputs between zero and one. Its output was useful for early classification, but learning can be slow in deep networks. section 3.1.4
  • Tanh centers its output around zero, supporting positive and negative signals. Its smoothness can also limit its usefulness in very deep models. section 3.1.4
  • ReLU outputs zero for negative inputs and passes positive inputs through linearly. It helped training signals flow through many layers. section 3.1.4
  • GELU scales inputs smoothly: small or negative signals can still influence the output, while stronger positive signals pass through more fully. It is common in modern deep networks, including Transformer-based language models. section 3.1.4
Figure 3.4 The ReLU activation function transforms a neuron’s input by suppressing negative values and passing positive ones through unchanged. This simple nonlinear behavior changes how inputs are separated, enabling the network to move beyond straight-line boundaries and capture more complex patterns.

ReLU illustrates how a simple nonlinear response can help a network move beyond straight-line boundaries. section 3.1.4

How Machines that Think covers it

How deep it goes
Explains the neuron pipeline and compares activation functions by response and training limitations.
Examples
Compares sigmoid, tanh, ReLU, and GELU as ways to shape a neuron's response.
What you can do after
Evaluate/understand how activation functions affect signal flow and network expressiveness.
How it connects
Connects activation functions to artificial neurons, weights, bias, and linear transformations.

Also explained in

Compare how 20 books explain activation function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.