ReLU
ReLU is an activation function that maps an input x to the greater of x and zero.
Grokking Machine Learning
An activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Uses ReLU between neural-network layers to break linearity and help build more complex decision boundaries.
- How deep it goes
- Defines ReLU with a piecewise rule and explains its positive-input derivative and connection to vanishing gradients.
- Examples
- Shows ReLU in Keras classifiers for a circular dataset and handwritten digits.
- What you can do after
- Apply it via Keras by assigning ReLU activations to hidden layers in a classifier.
Build AI Drug Discovery Pipelines
ReLU is an activation function that outputs zero for negative inputs and otherwise returns the input value.
- How it is used
- Introduces nonlinearity into drug-discovery neural networks for molecular predictions and graph representations.
- How deep it goes
- Explains activation behavior and gradient motivation, then demonstrates placement in network forward passes.
- Examples
- Uses a kinase binder network and a molecular-fingerprint surrogate predicting docking scores.
- What you can do after
- Apply it via PyTorch using nn.ReLU() between neural network layers.
Inside Deep Learning: Math, Algorithms, Models
A rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Deploys ReLU as a hidden-layer activation to improve convergence and accuracy in feed-forward and convolutional networks.
- How deep it goes
- Explains the piecewise function, derivative, gradient behavior, plots, and LeakyReLU tradeoff, but defers broader architecture design.
- Examples
- Uses NumPy activation values, comparison plots, and Fashion-MNIST fully connected and CNN training experiments.
- What you can do after
- Apply ReLU by replacing saturating hidden-layer activations and diagnose inactive-neuron risks through derivative and training behavior.
Read how Inside Deep Learning: Math, Algorithms, Models explains ReLU →
Sutskever's List
ReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Enables effective training at AlexNet’s depth by reducing gradient loss associated with saturating activations.
- How deep it goes
- Defines the input-output rule and explains positive-side gradient flow, then addresses inactive units through initialization rather than a formal derivation.
- Examples
- Compares activation-function plots and reports AlexNet’s sixfold convergence improvement when replacing tanh with ReLU.
- What you can do after
- Evaluate/understand it as a training choice, including why initialization matters when ReLU units risk remaining inactive.
Grokking Machine Learning, Second Edition
A rectified linear unit (ReLU) is an activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
- How it is used
- Uses ReLU between neural-network layers to add nonlinearity and improve training in classification and regression.
- How deep it goes
- Defines ReLU mathematically, explains its gradient advantage, and demonstrates it in Keras architectures without deriving implementation internals.
- Examples
- Demonstrates ReLU with a circular two-class dataset and a Hyderabad housing-price regression model.
- What you can do after
- Apply ReLU via Keras when configuring hidden layers for neural-network classification or regression.
Read how Grokking Machine Learning, Second Edition explains ReLU →
Deep Learning with R, Third Edition
A rectified linear unit is the activation function that maps an input x to max(x, 0).
- How it is used
- Uses ReLU in Dense-layer stacks to build nonlinear representations for problems such as sentiment classification.
- How deep it goes
- Defines ReLU, explains its tensor and geometric roles, and demonstrates implementation, while deferring deeper treatment of nonlinear transformations.
- Examples
- Demonstrates ReLU with R array code, optimized pmax operations, geometric figures, and a Keras sentiment-classification model.
- What you can do after
- Apply it via Keras or tensor operations to introduce nonlinear behavior in neural-network layers.
Read how Deep Learning with R, Third Edition explains ReLU →
LLMs in Production
A rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
- How it is used
- Places an activation function inside CBoW and small language-model training loops.
- How deep it goes
- Defines behavior and implementation placement, but assumes activation-function familiarity and defers a full explanation.
- Examples
- Uses a NumPy CBoW language model and a PyTorch two-layer feed-forward language model.
- What you can do after
- Build it from scratch in a training loop or place it between linear layers in a small language model.
Deep Learning with Python, Third Edition
A rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
- How it is used
- Uses ReLU as an intermediate activation in Dense neural networks, including sentiment classification.
- How deep it goes
- Defines ReLU as max(x, 0), then explains element-wise and vectorized execution without developing deeper activation theory.
- Examples
- Demonstrates ReLU with NumPy arrays, a naive Python loop, and a Keras Dense-layer model.
- What you can do after
- Apply it via NumPy or Keras when constructing Dense neural networks and selecting intermediate activations.
Read how Deep Learning with Python, Third Edition explains ReLU →
Machine Learning Algorithms in Depth
ReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
- How it is used
- Adds nonlinearity between affine layers in multilayer perceptrons and between convolution and pooling in image classifiers.
- How deep it goes
- Defines ReLU and places it in layer compositions and a CNN architecture; does not develop its derivative.
- Examples
- Uses a multilayer perceptron and a LeNet-style MNIST digit classifier with repeated convolution–ReLU–pooling stages.
- What you can do after
- Apply it via Keras/TensorFlow as the activation between convolution and max-pooling operations in an MNIST classifier.
Read how Machine Learning Algorithms in Depth explains ReLU →
Math and Architectures of Deep Learning
The rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
- How it is used
- Uses ReLU to add nonlinearity throughout VGG convolutional blocks and classifier layers.
- How deep it goes
- Defines ReLU mathematically, gives geometric intuition, and shows PyTorch placement; it does not detail gradient behavior.
- Examples
- Shows ReLU in VGG convolutional blocks, classifier layers, and the PyTorch nn.ReLU implementation.
- What you can do after
- Apply ReLU via PyTorch when constructing VGG-style convolutional blocks and classifiers.
Read how Math and Architectures of Deep Learning explains ReLU →
Deep Learning with PyTorch, Second Edition
A rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
- How it is used
- Uses ReLU between convolutional layers in a three-dimensional convolutional network block.
- How deep it goes
- Defines ReLU behavior, compares related activations, and demonstrates its placement in a hand-built convolutional block.
- Examples
- Shows ReLU in the LunaBlock code and compares it visually with common and less-common activation functions.
- What you can do after
- Apply ReLU through PyTorch's nn.ReLU layer inside a convolutional block.
Read how Deep Learning with PyTorch, Second Edition explains ReLU →
Data Without Labels
A rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
- How it is used
- Uses ReLU in hidden layers to train neural networks faster and add nonlinearity in CNNs.
- How deep it goes
- Defines the piecewise function and key computational properties, while deferring other activation functions.
- Examples
- Uses F(x)=max(0,x), a ReLU graph, and CNN processing after convolution extracts image features.
- What you can do after
- Evaluate ReLU as a fast hidden-layer activation and explain its role in modeling complex CNN patterns.
Learn Generative AI with PyTorch
A piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
- How it is used
- Uses ReLU as the default hidden-layer activation for introducing nonlinearity in PyTorch neural networks.
- How deep it goes
- Explains ReLU through weighted-sum decisions, nonlinear outputs, layer placement, and code, without deriving optimization behavior.
- Examples
- Demonstrates ReLU in an ankle-boot classifier and a two-dimensional discriminator network.
- What you can do after
- Apply ReLU via PyTorch's nn.ReLU in hidden layers of binary classifiers and discriminator networks.
Timeless Algorithms, The Foundational Ideas
A rectified linear unit (ReLU) is an activation function that outputs the greater of zero and its input.
- How it is used
- Shapes hidden-layer representations and supports learning; deep hidden layers often use ReLU because it preserves stronger gradients for positive inputs.
- How deep it goes
- Defines ReLU, compares outputs with sigmoid and tanh on two inputs, and notes its positive-input gradient advantage.
- Examples
- Uses pre-activation inputs 1.1 and 0.1 to compare ReLU outputs with sigmoid and tanh.
- What you can do after
- Evaluate how ReLU changes hidden representations relative to sigmoid and tanh, and understand its stated positive-input gradient advantage.
Read how Timeless Algorithms, The Foundational Ideas explains ReLU →
Grokking AI Algorithms, Second Edition
A ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.
- How it is used
- Uses ReLU to add nonlinear learning capacity and selectively deactivate neurons in artificial and language-model neural networks.
- How deep it goes
- Explains the elementwise rule and traces a vector example, but does not develop derivatives or optimization details.
- Examples
- Demonstrates ReLU on a five-value vector and places it after weight multiplication and bias addition in a projection layer.
- What you can do after
- Apply ReLU element by element after a neural-network layer’s weighted sum and bias addition.
Read how Grokking AI Algorithms, Second Edition explains ReLU →
CUDA for LLMs
ReLU is an activation function that returns the maximum of zero and its input.
- How it is used
- Uses ReLU as a neural-network activation and as an example of an element-wise GPU workload.
- How deep it goes
- Derives the forward function and backward gradient mask, then frames element-wise execution and fused-kernel design.
- Examples
- Illustrates ReLU on a small matrix and in the composite operation ReLU(LayerNorm(x)).
- What you can do after
- Build a parallelization strategy for a fused ReLU(LayerNorm(x)) kernel.
Also covered in
- Machines that ThinkIn early access (MEAP)
- Grokking GPU ProgrammingIn early access (MEAP)
What to know first
- classifierA classifier is a predictive model that assigns an input to one or more discrete classes or categories.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- linear regressionLinear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Related concepts
- activation functionkind of · An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.
- multilayer perceptronused by · A multilayer perceptron is a feedforward neural network composed of fully connected layers with nonlinear transformations.
- AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
- neural networkused by · A neural network is a computational model composed of interconnected, parameterized units or transformations that learn to map inputs to outputs.
- GPUused by · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
Questions about ReLU
What is ReLU?
ReLU is an activation function that maps an input x to the greater of x and zero.
Which Manning books explain ReLU?
Grokking Machine Learning, Build AI Drug Discovery Pipelines, Inside Deep Learning: Math, Algorithms, Models, Sutskever's List, Grokking Machine Learning, Second Edition, Deep Learning with R, Third Edition, LLMs in Production, Deep Learning with Python, Third Edition and 8 more.
What should I know before learning ReLU?
Manning books that teach ReLU usually introduce classifier, language model, linear regression and objective function first.
What is ReLU a kind of?
ReLU is a kind of activation function. An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
What is ReLU also called?
ReLU is also called rectified linear unit or ReLUs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.















