Activation function

As explained in Machine Learning Algorithms in Depth

An activation function is a nonlinear function applied to a neural-network layer output to increase the model's expressive capacity.

An activation function applies a nonlinearity between neural-network layers, allowing stacked affine transformations to represent more expressive functions than a purely linear network. section 10.1

Choose the activation according to the layer's role and prediction task:

• Use ReLU as a common hidden-layer nonlinearity in a multilayer perceptron. section 10.1
• Use softmax at a classification output to convert real-valued inputs into class probabilities summing to one. section 10.1
• Use the identity function at a regression output. section 10.1

Figure 10.1 MLP architecture

Use the MLP architecture figure to locate activation functions between affine layers and distinguish hidden-layer and output-layer roles. section 10.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines ReLU and identifies softmax and identity by output task, but defers detailed activation derivatives and comparisons.

Examples

Uses ReLU in hidden layers, softmax for classification, identity for regression, and an MLP architecture as concrete examples.

What you can do after

Evaluate activation choices for hidden layers and classification or regression output layers.

How it connects

Connects activation functions to multilayer perceptrons, affine transformations, softmax, classification, regression, loss functions, and backpropagation.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA function that transforms a model's score into an output or prediction.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionAn activation function is a nonlinear or linear mathematical function that transforms a neuron's weighted input before passing the result onward.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionAn activation function is a nonlinear function applied to a neural-network signal to transform its value before further processing.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmersAn activation function is a function applied to a neuron's output to introduce nonlinearity and constrain the output range in a neural network.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA nonlinear function applied in a neural network to transform layer outputs and increase representational power.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
• 

  liveBookmanning.com
  Data Without LabelsAn activation function is a mathematical function applied to a neuron's combined input to determine its output, typically introducing nonlinearity.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn activation function is a function applied to a neural-network unit’s pre-activation value to produce its output and introduce nonlinear behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
• 

  liveBookmanning.com
  Sutskever's ListAn activation function transforms a neuron's weighted input before passing its output to subsequent layers.
• 

  liveBookmanning.com
  Deep Learning with JAXA nonlinear function applied inside a neural network to let stacked layers represent more than a single linear transformation.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn activation function is a nonlinear or otherwise specified function applied to a layer's transformed values.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.

Compare how 14 books explain activation function →

Related concepts

• sigmoidincludes · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
• ReLUincludes · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• nonlinear mappingproduces · A nonlinear mapping is a function that does not preserve linear combinations of its inputs.
• step functionincludes · A step function is a piecewise-constant function that changes abruptly between discrete output values when its input crosses one or more specified thresholds.
• GELUincludes · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
• Swishincludes · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.