How deeply does Math and Architectures of Deep Learning cover ReLU?

Defines ReLU mathematically, gives geometric intuition, and shows PyTorch placement; it does not detail gradient behavior.

ReLU

As explained in Math and Architectures of Deep Learning

The rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.

ReLU, or the rectified linear unit, maps an input x to max(0, x). It outputs zero for negative inputs and preserves positive inputs, adding nonlinearity after convolutional layers. In VGG, repeated ReLU activations make stacked 3 × 3 convolutions more expressive. section 11.2.1

Its practical behavior in convolutional networks includes:

Negative activations become zero, while positive activations pass through unchanged. section 11.2.1
VGG places a ReLU after each convolution, increasing nonlinearity without changing the convolution's spatial resolution. section 11.2.1
ReLU is unbounded; unlike local response normalization, it does not bound large outputs. section 11.2.1

PyTorch inside a convolutional block

A VGG-style block pairs each 3 × 3 convolution with nn.ReLU(inplace=True), then applies max pooling after the block. The same activation also appears in the fully connected classifier. section 11.2.1

Equation 11.1

ReLU(x) = max(0, x)

Use the equation to verify the zero-for-negative and identity-for-positive behavior. section 11.2.1

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Grokking Machine LearningAn activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
liveBook manning.com

Sutskever's ListReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
liveBook manning.com

Deep Learning with R, Third EditionA rectified linear unit is the activation function that maps an input x to max(x, 0).
liveBook manning.com

LLMs in ProductionA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
liveBook manning.com

Deep Learning with Python, Third EditionA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
liveBook manning.com

Machine Learning Algorithms in DepthReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
liveBook manning.com

Data Without LabelsA rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
liveBook manning.com

Learn Generative AI with PyTorchA piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
liveBook manning.com

Grokking AI Algorithms, Second EditionA ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

Compare how 12 books explain ReLU →

Related concepts

activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.