ReLU

As explained in Data Without Labels

A rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.

A rectified linear unit (ReLU) returns its input when positive and returns zero otherwise: F(x) = max(0, x). Its simple, unbounded form is inexpensive to compute, though it is not centered at zero and is not differentiable at zero. section 8.2.4

ReLU is widely used in hidden layers because its low computational cost can speed network training. In convolutional neural networks, it adds nonlinearity so the model can represent complex patterns after convolution extracts features. section 8.2.4 section 8.6.1

(8.3)

F(x) = max (0, x)

Use the equation to inspect ReLU’s piecewise behavior: negative inputs map to zero, while positive inputs pass through unchanged. section 8.2.4

Figure 8.7 A ReLU function. It is one of the favored activation functions in the hidden layers of a neural network. A ReLU is simple to use and less expensive to train.

Use the graph to see ReLU’s flat nonpositive region and rising positive region, as well as its role in hidden layers. section 8.2.4

How Data Without Labels covers it

How deep it goes

Defines the piecewise function and key computational properties, while deferring other activation functions.

Examples

Uses F(x)=max(0,x), a ReLU graph, and CNN processing after convolution extracts image features.

What you can do after

Evaluate ReLU as a fast hidden-layer activation and explain its role in modeling complex CNN patterns.

How it connects

Connects ReLU with activation functions, hidden layers, convolution, pooling, and softmax output layers.

liveBookmanning.com

Read Data Without Labels on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Sutskever's ListReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA rectified linear unit is the activation function that maps an input x to max(x, 0).
• 

  liveBookmanning.com
  LLMs in ProductionA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

Compare how 12 books explain ReLU →

Related concepts

• activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
• tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
• Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
• SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

This explainer was generated by Manning's knowledge graph from the text of Data Without Labels. The section links point to the parts of the book it draws on.