ReLU
ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
Grokking Machine Learning
An activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Uses ReLU between neural-network layers to break linearity and help build more complex decision boundaries.
- How deep it goes
- Defines ReLU with a piecewise rule and explains its positive-input derivative and connection to vanishing gradients.
- Examples
- Shows ReLU in Keras classifiers for a circular dataset and handwritten digits.
- What you can do after
- Apply it via Keras by assigning ReLU activations to hidden layers in a classifier.
Inside Deep Learning: Math, Algorithms, Models
A rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Deploys ReLU as a hidden-layer activation to improve convergence and accuracy in feed-forward and convolutional networks.
- How deep it goes
- Explains the piecewise function, derivative, gradient behavior, plots, and LeakyReLU tradeoff, but defers broader architecture design.
- Examples
- Uses NumPy activation values, comparison plots, and Fashion-MNIST fully connected and CNN training experiments.
- What you can do after
- Apply ReLU by replacing saturating hidden-layer activations and diagnose inactive-neuron risks through derivative and training behavior.
Read how Inside Deep Learning: Math, Algorithms, Models explains ReLU →
Sutskever's List
ReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Enables effective training at AlexNet’s depth by reducing gradient loss associated with saturating activations.
- How deep it goes
- Defines the input-output rule and explains positive-side gradient flow, then addresses inactive units through initialization rather than a formal derivation.
- Examples
- Compares activation-function plots and reports AlexNet’s sixfold convergence improvement when replacing tanh with ReLU.
- What you can do after
- Evaluate/understand it as a training choice, including why initialization matters when ReLU units risk remaining inactive.
Deep Learning with R, Third Edition
A rectified linear unit is the activation function that maps an input x to max(x, 0).
- How it is used
- Uses ReLU in Dense-layer stacks to build nonlinear representations for problems such as sentiment classification.
- How deep it goes
- Defines ReLU, explains its tensor and geometric roles, and demonstrates implementation, while deferring deeper treatment of nonlinear transformations.
- Examples
- Demonstrates ReLU with R array code, optimized pmax operations, geometric figures, and a Keras sentiment-classification model.
- What you can do after
- Apply it via Keras or tensor operations to introduce nonlinear behavior in neural-network layers.
Read how Deep Learning with R, Third Edition explains ReLU →
LLMs in Production
A rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
- How it is used
- Places an activation function inside CBoW and small language-model training loops.
- How deep it goes
- Defines behavior and implementation placement, but assumes activation-function familiarity and defers a full explanation.
- Examples
- Uses a NumPy CBoW language model and a PyTorch two-layer feed-forward language model.
- What you can do after
- Build it from scratch in a training loop or place it between linear layers in a small language model.
Deep Learning with Python, Third Edition
A rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
- How it is used
- Uses ReLU as an intermediate activation in Dense neural networks, including sentiment classification.
- How deep it goes
- Defines ReLU as max(x, 0), then explains element-wise and vectorized execution without developing deeper activation theory.
- Examples
- Demonstrates ReLU with NumPy arrays, a naive Python loop, and a Keras Dense-layer model.
- What you can do after
- Apply it via NumPy or Keras when constructing Dense neural networks and selecting intermediate activations.
Read how Deep Learning with Python, Third Edition explains ReLU →
Machine Learning Algorithms in Depth
ReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
- How it is used
- Adds nonlinearity between affine layers in multilayer perceptrons and between convolution and pooling in image classifiers.
- How deep it goes
- Defines ReLU and places it in layer compositions and a CNN architecture; does not develop its derivative.
- Examples
- Uses a multilayer perceptron and a LeNet-style MNIST digit classifier with repeated convolution–ReLU–pooling stages.
- What you can do after
- Apply it via Keras/TensorFlow as the activation between convolution and max-pooling operations in an MNIST classifier.
Read how Machine Learning Algorithms in Depth explains ReLU →
Math and Architectures of Deep Learning
The rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
- How it is used
- Uses ReLU to add nonlinearity throughout VGG convolutional blocks and classifier layers.
- How deep it goes
- Defines ReLU mathematically, gives geometric intuition, and shows PyTorch placement; it does not detail gradient behavior.
- Examples
- Shows ReLU in VGG convolutional blocks, classifier layers, and the PyTorch nn.ReLU implementation.
- What you can do after
- Apply ReLU via PyTorch when constructing VGG-style convolutional blocks and classifiers.
Read how Math and Architectures of Deep Learning explains ReLU →
Deep Learning with PyTorch, Second Edition
A rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
- How it is used
- Uses ReLU between convolutional layers in a three-dimensional convolutional network block.
- How deep it goes
- Defines ReLU behavior, compares related activations, and demonstrates its placement in a hand-built convolutional block.
- Examples
- Shows ReLU in the LunaBlock code and compares it visually with common and less-common activation functions.
- What you can do after
- Apply ReLU through PyTorch's nn.ReLU layer inside a convolutional block.
Read how Deep Learning with PyTorch, Second Edition explains ReLU →
Data Without Labels
A rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
- How it is used
- Uses ReLU in hidden layers to train neural networks faster and add nonlinearity in CNNs.
- How deep it goes
- Defines the piecewise function and key computational properties, while deferring other activation functions.
- Examples
- Uses F(x)=max(0,x), a ReLU graph, and CNN processing after convolution extracts image features.
- What you can do after
- Evaluate ReLU as a fast hidden-layer activation and explain its role in modeling complex CNN patterns.
Learn Generative AI with PyTorch
A piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
- How it is used
- Uses ReLU as the default hidden-layer activation for introducing nonlinearity in PyTorch neural networks.
- How deep it goes
- Explains ReLU through weighted-sum decisions, nonlinear outputs, layer placement, and code, without deriving optimization behavior.
- Examples
- Demonstrates ReLU in an ankle-boot classifier and a two-dimensional discriminator network.
- What you can do after
- Apply ReLU via PyTorch's nn.ReLU in hidden layers of binary classifiers and discriminator networks.
Grokking AI Algorithms, Second Edition
A ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.
- How it is used
- Uses ReLU to add nonlinear learning capacity and selectively deactivate neurons in artificial and language-model neural networks.
- How deep it goes
- Explains the elementwise rule and traces a vector example, but does not develop derivatives or optimization details.
- Examples
- Demonstrates ReLU on a five-value vector and places it after weight multiplication and bias addition in a projection layer.
- What you can do after
- Apply ReLU element by element after a neural-network layer’s weighted sum and bias addition.
Read how Grokking AI Algorithms, Second Edition explains ReLU →
Related concepts
- activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
- AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
- tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
- Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
- SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.