ReLU

As explained in Grokking Machine Learning

An activation function that returns zero for negative inputs and the input itself for nonnegative inputs.

ReLU, or rectified linear unit, leaves nonnegative inputs unchanged and turns negative inputs into zero: ReLU(x) = x for x ≥ 0, and ReLU(x) = 0 for x < 0. Used between neural-network layers, it helps break linearity and build more complex decision boundaries. section 10.2 section 10.6

For positive inputs, ReLU’s derivative is 1, which helps reduce the vanishing-gradient problem. It is widely used in neural networks, in contrast to sigmoid and tanh, whose flat derivatives can contribute to that problem. section 10.2 section 10.6

ReLU in a Keras network

A classifier for a circular dataset uses ReLU in hidden layers of 128 and 64 units, followed by a two-unit softmax output. A handwritten-digit classifier uses the same hidden-layer sizes and ReLU activations. section 10.3

Figure 10.19 Three different activation functions used in neural networks. Left: The sigmoid function, represented by the Greek letter sigma. Middle: The hyperbolic tangent, or tanh. Right: The rectified linear unit, or ReLU.

Compare the plotted shapes of sigmoid, tanh, and ReLU. section 10.2

How Grokking Machine Learning covers it

How deep it goes

Defines ReLU with a piecewise rule and explains its positive-input derivative and connection to vanishing gradients.

Examples

Shows ReLU in Keras classifiers for a circular dataset and handwritten digits.

What you can do after

Apply it via Keras by assigning ReLU activations to hidden layers in a classifier.

How it connects

Connects ReLU to neural networks, perceptrons, activation functions, sigmoid, tanh, and vanishing gradients.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Sutskever's ListReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA rectified linear unit is the activation function that maps an input x to max(x, 0).
• 

  liveBookmanning.com
  LLMs in ProductionA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Data Without LabelsA rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

Compare how 12 books explain ReLU →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
• tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
• Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
• SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.