Activation function

As explained in Grokking Machine Learning

A function that transforms a model's score into an output or prediction.

When a model calculates a score, it still needs a rule for turning that score into an output. An activation function performs that transformation. In a perceptron, the step function returns 1 when the score is nonnegative and 0 when it is negative: step(x) = 1 if x ≥ 0, otherwise 0. Thus, ŷ = step(ax₁ + bx₂ + c) converts a weighted score into a prediction. section 5.1

Common activation functions shape model outputs in different ways:

• The step function makes a hard binary decision. section 5.1
• The sigmoid function maps any real number to a value between 0 and 1, supporting soft assignments such as a 60% or 95% happiness score. section 6.0
• Sigmoid, hyperbolic tangent, softmax, and ReLU are used in neural networks; activation functions between layers break linearity and enable more complex boundaries. section 10.6
• ReLU sends negative numbers to 0 and leaves nonnegative numbers unchanged, helping reduce the vanishing-gradient problem during training. section 10.6

Classifying an alien's mood

Represent a sentence by counting words such as “aack” and “beep.” A perceptron computes a score from those counts, weights, and a bias. Applying the step function labels the sentence happy when the score is at least zero and sad otherwise. A logistic classifier replaces this abrupt decision with a score between 0 and 1, allowing sentences to be compared by how happy they appear. section 5.1 section 6.0

Figure 5.8 The step function is useful in the study of perceptron models. The output of the step function is 0 when the input is negative and 1 otherwise.

Use the graph to see how the step function jumps from 0 for negative inputs to 1 for nonnegative inputs. section 5.1

How Grokking Machine Learning covers it

How deep it goes

Defines threshold and continuous functions, compares their outputs, and connects them to neural-network nonlinearity and gradient problems.

Examples

Uses alien mood classification, word-count features, perceptron scores, sigmoid happiness ratings, and neural-network layers.

What it assumes

Assumes weighted scores and perceptrons, while developing the distinction between hard and soft classification from basic examples.

What you can do after

Apply a named activation function to turn weighted model scores into binary predictions, continuous scores, or nonlinear neural-network outputs.

How it connects

Connects activation functions to perceptrons, logistic classifiers, neural networks, classification, regression, backpropagation, and vanishing gradients.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionAn activation function is a nonlinear or linear mathematical function that transforms a neuron's weighted input before passing the result onward.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionAn activation function is a nonlinear function applied to a neural-network signal to transform its value before further processing.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmersAn activation function is a function applied to a neuron's output to introduce nonlinearity and constrain the output range in a neural network.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA nonlinear function applied in a neural network to transform layer outputs and increase representational power.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
• 

  liveBookmanning.com
  Data Without LabelsAn activation function is a mathematical function applied to a neuron's combined input to determine its output, typically introducing nonlinearity.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn activation function is a function applied to a neural-network unit’s pre-activation value to produce its output and introduce nonlinear behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
• 

  liveBookmanning.com
  Sutskever's ListAn activation function transforms a neuron's weighted input before passing its output to subsequent layers.
• 

  liveBookmanning.com
  Deep Learning with JAXA nonlinear function applied inside a neural network to let stacked layers represent more than a single linear transformation.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn activation function is a nonlinear or otherwise specified function applied to a layer's transformed values.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAn activation function is a nonlinear function applied to a neural-network layer output to increase the model's expressive capacity.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.

Compare how 14 books explain activation function →

Related concepts

• sigmoidincludes · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
• ReLUincludes · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• nonlinear mappingproduces · A nonlinear mapping is a function that does not preserve linear combinations of its inputs.
• step functionincludes · A step function is a piecewise-constant function that changes abruptly between discrete output values when its input crosses one or more specified thresholds.
• GELUincludes · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
• Swishincludes · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.