liveBook · Manning

Activation function

As explained in Timeless Algorithms, The Foundational Ideas

An activation function transforms a neural network unit’s combined inputs before the result is passed onward.

An activation function transforms a neural network unit’s combined inputs before passing the result onward. Its nonlinearity allows successive layers to represent relationships that repeated linear transformations alone cannot capture. section 11.2.1 section 11.8

For inputs x₁, …, xₙ, weights w₁, …, wₙ, and bias b, a unit first computes z = w₁x₁ + … + wₙxₙ + b. Its activation is then a = ϕ(z), where ϕ is the activation function. The value z is the pre-activation; a becomes an input to the next layer. Weights determine the influence of each input, while the bias shifts the unit’s response. section 11.2.2 section 11.3.2 section 11.2.1

Each layer’s outputs form a representation: a numerical description shaped by its weights, biases, and activation functions. Geometrically, weighted combinations reorient and shift the data, while nonlinear activations introduce bends. Successive layers can build progressively more useful descriptions for prediction. section 11.2.1 section 11.2.2

Different activation functions transform the same pre-activation in different ways:

  • Sigmoid maps inputs to values between 0 and 1; zero maps to 0.5. section 11.4.1
  • Tanh maps inputs to values between −1 and 1 and is centered at zero. Positive and negative inputs produce outputs with the same respective signs. section 11.4.1
  • ReLU is defined as max(0, z): it sets negative inputs to zero and passes positive inputs through unchanged. section 11.4.1

Comparing hidden-node activations

Suppose two hidden nodes receive the same input, x = 2.0. With weights and biases of 0.8 and −0.5 for the first node, and −0.4 and 0.9 for the second, their pre-activations are 1.1 and 0.1. Sigmoid produces approximately 0.75 and 0.52; tanh produces 0.80 and 0.10; ReLU produces 1.10 and 0.10. Thus the activation choice changes what the next layer receives even when the pre-activations are unchanged. section 11.3.2 section 11.4.1

A value between 0 and 1 is not automatically a probability. In the example, hidden-node sigmoid outputs are internal signals; the output-node sigmoid value represents the predicted probability of the positive class. Activation choice also changes gradients during learning, so it affects both representation and training. section 11.3.2 section 11.4.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through weighted inputs and compares activation outputs, relating the choice to representations and gradients.
Examples
Compares sigmoid, tanh, and ReLU on two hidden-node pre-activations derived from an image-based braking scenario.
What you can do after
Evaluate how activation choices change internal representations and gradients during learning.
How it connects
Connects activation functions to representation learning, logistic regression, and backpropagation.

Also explained in

Compare how 20 books explain activation function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.