ReLU

As explained in Machine Learning Algorithms in Depth

ReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.

A stack of affine layers alone is limited in expressive power. ReLU (rectified linear unit) adds a nonlinearity after a layer’s weighted sum: ReLU(z) = max(0, z). It outputs zero for negative inputs and leaves positive inputs unchanged. section 10.1

For a hidden layer, compute z = Wx + b, then pass ReLU(z) to the next layer. In a multilayer perceptron, the final activation instead depends on the task: softmax for classification or identity for regression. section 10.1

ReLU in digit classification

A LeNet-style MNIST classifier repeats convolution → ReLU → max pooling. Pooling reduces feature-map dimensions while later convolutional layers use more filters; the resulting features feed a dense layer and softmax classifier. section 10.2.1

Figure 10.4 LeNet architecture for MNIST digits classification

The LeNet architecture shows where ReLU sits between convolution and max pooling. section 10.2.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines ReLU and places it in layer compositions and a CNN architecture; does not develop its derivative.

Examples

Uses a multilayer perceptron and a LeNet-style MNIST digit classifier with repeated convolution–ReLU–pooling stages.

What it assumes

Assumes familiarity with weighted sums and layers while introducing why activation functions are needed.

What you can do after

Apply it via Keras/TensorFlow as the activation between convolution and max-pooling operations in an MNIST classifier.

How it connects

Connects ReLU to affine layers, multilayer perceptrons, convolutional layers, max pooling, and backpropagation.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Sutskever's ListReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA rectified linear unit is the activation function that maps an input x to max(x, 0).
• 

  liveBookmanning.com
  LLMs in ProductionA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Data Without LabelsA rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

Compare how 12 books explain ReLU →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
• tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
• Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
• SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.