liveBook · Manning

Activation function

As explained in Fight Fraud with Machine Learning

An activation function is a function applied to a neural-network unit's weighted input to determine its output and introduce nonlinear behavior.

An activation function transforms a neuron's weighted input before producing its output. Its nonlinearity lets stacked neural-network layers learn complex patterns and decision boundaries; without it, multiple layers collapse into one weighted-sum layer. section 4.1.1

Common activation functions serve different modeling needs:

  • Sigmoid and TanH squash inputs into bounded ranges, making them useful for binary classification or normalized outputs. section 4.1.1
  • ReLU offers faster computation and helps mitigate the vanishing-gradient problem associated with Sigmoid and TanH toward their extreme ranges. section 4.1.1
  • The step function provides the perceptron's earlier activation approach, while Sigmoid, TanH, and ReLU are more evolved choices for deep networks. section 4.1.1

Why nonlinearity matters

Consider several hidden layers that each compute weighted sums. If every layer remains linear, multiplying the weight matrices produces an equivalent single layer. Applying an activation function between layers prevents that collapse and enables richer representations for fraud-detection models. section 4.1.1

Figure 4.5 Some examples of activation functions commonly used in deep neural networks – Sigmoid, TanH, and ReLU.

Compare the shapes and ranges of Sigmoid, TanH, and ReLU when selecting an activation function. section 4.1.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains weighted inputs, layer collapse, expressive power, bounded outputs, computation, and vanishing gradients.
Examples
Compares the perceptron step function with Sigmoid, TanH, and ReLU through an activation-function figure.
What it assumes
Builds from perceptrons and weighted sums before extending the explanation to hidden layers and deep neural networks.
What you can do after
Evaluate activation choices by relating bounded outputs, computation speed, nonlinearity, and vanishing-gradient behavior to model needs.
How it connects
Connects perceptrons, weighted sums, hidden layers, deep neural networks, gradient descent, and vanishing gradients.

Also explained in

Compare how 20 books explain activation function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.