ReLU

As explained in Sutskever's List

ReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.

AlexNet faced a practical obstacle: deep networks were difficult to train when sigmoid or tanh activations saturated and gradients dwindled. Its adoption of the rectified linear unit, or ReLU, helped gradients pass through many layers and made larger experiments feasible. section 2.5.2

ReLU outputs zero for a negative input and leaves a nonnegative input unchanged. For example, −2 becomes 0, while 3 remains 3. On the positive side, its constant derivative lets a gradient pass without the repeated shrinking associated with saturating activations. section 2.5.2

Figure 2.4 Plot of ReLU (top), sigmoid (middle), and tanh (bottom) activation functions generated in R by the author. Activation functions map input values (on the x-axis) to output values (on the y-axis). For example, ReLU produces 0 for negative inputs; for positive inputs, the output equals the input (0 for x < 0 and linear for x ≥ 0). The sigmoid function maps inputs to values between 0 and 1, resembling an S-curve. Similarly, the tanh function provides a smooth S-shaped curve but maps inputs to values between –1 and 1, centered at 0.

Compare ReLU’s flat negative side and rising positive side with the bounded curves of sigmoid and tanh. section 2.5.2

In a controlled comparison, switching from tanh to ReLU accelerated AlexNet’s convergence by a factor of six. ReLU was not a complete training recipe, however: inactive units could become stuck outputting zero. AlexNet set biases in specific layers to 1 to activate ReLU neurons at the start of training. section 2.5.2 section 2.5.4

ReLU became a default activation for deep networks and appeared, or was adapted, in later architectures including VGGNet, GoogLeNet, and ResNet. AlexNet’s result also depended on weight initialization, momentum, dropout, data augmentation, and GPU engineering. Still deeper networks continued to face vanishing gradients and model degradation. section 2.5.2 section 2.5.5

How Sutskever's List covers it

How deep it goes

Defines the input-output rule and explains positive-side gradient flow, then addresses inactive units through initialization rather than a formal derivation.

Examples

Compares activation-function plots and reports AlexNet’s sixfold convergence improvement when replacing tanh with ReLU.

What it assumes

Recaps how activation functions map inputs to outputs while relying on neural-network layers and training gradients for context.

What you can do after

Evaluate/understand it as a training choice, including why initialization matters when ReLU units risk remaining inactive.

How it connects

Links ReLU to sigmoid, tanh, vanishing gradients, weight initialization, and deep convolutional networks.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA rectified linear unit is the activation function that maps an input x to max(x, 0).
• 

  liveBookmanning.com
  LLMs in ProductionA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Data Without LabelsA rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

Compare how 12 books explain ReLU →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
• tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
• Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
• SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.