Activation function

As explained in Sutskever's List

An activation function transforms a neuron's weighted input before passing its output to subsequent layers.

An activation function transforms a neuron's weighted input before it passes the result onward. In AlexNet, ReLU followed each convolutional layer, helping the network train at greater depth. section 2.5.1 section 2.5.2

Activation functions differ in how they transform inputs:

• ReLU maps negative inputs to zero and leaves positive inputs unchanged, supporting stronger gradient flow through deep networks. section 2.5.2
• Sigmoid and tanh compress outputs into limited ranges; saturation can contribute to vanishing gradients. section 2.5.2

ReLU versus tanh in AlexNet

In a controlled comparison, replacing tanh with ReLU accelerated convergence sixfold. That efficiency allowed experiments with larger networks, helping establish ReLU as a default for deep networks. section 2.5.2

Figure 2.4 Plot of ReLU (top), sigmoid (middle), and tanh (bottom) activation functions generated in R by the author. Activation functions map input values (on the x-axis) to output values (on the y-axis). For example, ReLU produces 0 for negative inputs; for positive inputs, the output equals the input (0 for x < 0 and linear for x ≥ 0). The sigmoid function maps inputs to values between 0 and 1, resembling an S-curve. Similarly, the tanh function provides a smooth S-shaped curve but maps inputs to values between –1 and 1, centered at 0.

Compare the input-output curves for ReLU, sigmoid, and tanh. section 2.5.2

How Sutskever's List covers it

How deep it goes

Explains ReLU's input mapping, contrasts it with sigmoid and tanh, and connects gradient flow to convergence without formal derivation.

Examples

Compares ReLU, sigmoid, and tanh curves and reports a sixfold convergence gain over tanh in AlexNet.

What it assumes

Recaps neurons as computational blocks that apply weights and pass results through activation functions.

What you can do after

Evaluate how ReLU's input mapping and gradient behavior differ from sigmoid and tanh.

How it connects

Connects activation functions to neurons, gradient flow, vanishing gradients, and deep neural networks.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA function that transforms a model's score into an output or prediction.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionAn activation function is a nonlinear or linear mathematical function that transforms a neuron's weighted input before passing the result onward.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionAn activation function is a nonlinear function applied to a neural-network signal to transform its value before further processing.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmersAn activation function is a function applied to a neuron's output to introduce nonlinearity and constrain the output range in a neural network.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA nonlinear function applied in a neural network to transform layer outputs and increase representational power.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
• 

  liveBookmanning.com
  Data Without LabelsAn activation function is a mathematical function applied to a neuron's combined input to determine its output, typically introducing nonlinearity.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn activation function is a function applied to a neural-network unit’s pre-activation value to produce its output and introduce nonlinear behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
• 

  liveBookmanning.com
  Deep Learning with JAXA nonlinear function applied inside a neural network to let stacked layers represent more than a single linear transformation.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn activation function is a nonlinear or otherwise specified function applied to a layer's transformed values.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAn activation function is a nonlinear function applied to a neural-network layer output to increase the model's expressive capacity.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.

Compare how 14 books explain activation function →

Related concepts

• sigmoidincludes · A sigmoid is a smooth, bounded, S-shaped function that maps real-valued inputs between two limiting values, commonly zero and one.
• ReLUincludes · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• nonlinear mappingproduces · A nonlinear mapping is a function that does not preserve linear combinations of its inputs.
• step functionincludes · A step function is a piecewise-constant function that changes abruptly between discrete output values when its input crosses one or more specified thresholds.
• GELUincludes · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
• Swishincludes · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.