liveBook · Manning

ReLU

Also called rectified linear unit, ReLUs

ReLU is an activation function that maps an input x to the greater of x and zero.

How 16 Manning books explain it

Grokking Machine Learning

An activation function that returns zero for negative inputs and the input itself for nonnegative inputs.

How it is used
Uses ReLU between neural-network layers to break linearity and help build more complex decision boundaries.
How deep it goes
Defines ReLU with a piecewise rule and explains its positive-input derivative and connection to vanishing gradients.
Examples
Shows ReLU in Keras classifiers for a circular dataset and handwritten digits.
What you can do after
Apply it via Keras by assigning ReLU activations to hidden layers in a classifier.

Read how Grokking Machine Learning explains ReLU →

Build AI Drug Discovery Pipelines

ReLU is an activation function that outputs zero for negative inputs and otherwise returns the input value.

How it is used
Introduces nonlinearity into drug-discovery neural networks for molecular predictions and graph representations.
How deep it goes
Explains activation behavior and gradient motivation, then demonstrates placement in network forward passes.
Examples
Uses a kinase binder network and a molecular-fingerprint surrogate predicting docking scores.
What you can do after
Apply it via PyTorch using nn.ReLU() between neural network layers.

Read how Build AI Drug Discovery Pipelines explains ReLU →

Inside Deep Learning: Math, Algorithms, Models

A rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.

How it is used
Deploys ReLU as a hidden-layer activation to improve convergence and accuracy in feed-forward and convolutional networks.
How deep it goes
Explains the piecewise function, derivative, gradient behavior, plots, and LeakyReLU tradeoff, but defers broader architecture design.
Examples
Uses NumPy activation values, comparison plots, and Fashion-MNIST fully connected and CNN training experiments.
What you can do after
Apply ReLU by replacing saturating hidden-layer activations and diagnose inactive-neuron risks through derivative and training behavior.

Read how Inside Deep Learning: Math, Algorithms, Models explains ReLU →

Sutskever's List

ReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.

How it is used
Enables effective training at AlexNet’s depth by reducing gradient loss associated with saturating activations.
How deep it goes
Defines the input-output rule and explains positive-side gradient flow, then addresses inactive units through initialization rather than a formal derivation.
Examples
Compares activation-function plots and reports AlexNet’s sixfold convergence improvement when replacing tanh with ReLU.
What you can do after
Evaluate/understand it as a training choice, including why initialization matters when ReLU units risk remaining inactive.

Read how Sutskever's List explains ReLU →

Grokking Machine Learning, Second Edition

A rectified linear unit (ReLU) is an activation function that returns zero for negative inputs and the input itself for nonnegative inputs.

How it is used
Uses ReLU between neural-network layers to add nonlinearity and improve training in classification and regression.
How deep it goes
Defines ReLU mathematically, explains its gradient advantage, and demonstrates it in Keras architectures without deriving implementation internals.
Examples
Demonstrates ReLU with a circular two-class dataset and a Hyderabad housing-price regression model.
What you can do after
Apply ReLU via Keras when configuring hidden layers for neural-network classification or regression.

Read how Grokking Machine Learning, Second Edition explains ReLU →

Deep Learning with R, Third Edition

A rectified linear unit is the activation function that maps an input x to max(x, 0).

How it is used
Uses ReLU in Dense-layer stacks to build nonlinear representations for problems such as sentiment classification.
How deep it goes
Defines ReLU, explains its tensor and geometric roles, and demonstrates implementation, while deferring deeper treatment of nonlinear transformations.
Examples
Demonstrates ReLU with R array code, optimized pmax operations, geometric figures, and a Keras sentiment-classification model.
What you can do after
Apply it via Keras or tensor operations to introduce nonlinear behavior in neural-network layers.

Read how Deep Learning with R, Third Edition explains ReLU →

LLMs in Production

A rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.

How it is used
Places an activation function inside CBoW and small language-model training loops.
How deep it goes
Defines behavior and implementation placement, but assumes activation-function familiarity and defers a full explanation.
Examples
Uses a NumPy CBoW language model and a PyTorch two-layer feed-forward language model.
What you can do after
Build it from scratch in a training loop or place it between linear layers in a small language model.

Read how LLMs in Production explains ReLU →

Deep Learning with Python, Third Edition

A rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.

How it is used
Uses ReLU as an intermediate activation in Dense neural networks, including sentiment classification.
How deep it goes
Defines ReLU as max(x, 0), then explains element-wise and vectorized execution without developing deeper activation theory.
Examples
Demonstrates ReLU with NumPy arrays, a naive Python loop, and a Keras Dense-layer model.
What you can do after
Apply it via NumPy or Keras when constructing Dense neural networks and selecting intermediate activations.

Read how Deep Learning with Python, Third Edition explains ReLU →

Machine Learning Algorithms in Depth

ReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.

How it is used
Adds nonlinearity between affine layers in multilayer perceptrons and between convolution and pooling in image classifiers.
How deep it goes
Defines ReLU and places it in layer compositions and a CNN architecture; does not develop its derivative.
Examples
Uses a multilayer perceptron and a LeNet-style MNIST digit classifier with repeated convolution–ReLU–pooling stages.
What you can do after
Apply it via Keras/TensorFlow as the activation between convolution and max-pooling operations in an MNIST classifier.

Read how Machine Learning Algorithms in Depth explains ReLU →

Math and Architectures of Deep Learning

The rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.

How it is used
Uses ReLU to add nonlinearity throughout VGG convolutional blocks and classifier layers.
How deep it goes
Defines ReLU mathematically, gives geometric intuition, and shows PyTorch placement; it does not detail gradient behavior.
Examples
Shows ReLU in VGG convolutional blocks, classifier layers, and the PyTorch nn.ReLU implementation.
What you can do after
Apply ReLU via PyTorch when constructing VGG-style convolutional blocks and classifiers.

Read how Math and Architectures of Deep Learning explains ReLU →

Deep Learning with PyTorch, Second Edition

A rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.

How it is used
Uses ReLU between convolutional layers in a three-dimensional convolutional network block.
How deep it goes
Defines ReLU behavior, compares related activations, and demonstrates its placement in a hand-built convolutional block.
Examples
Shows ReLU in the LunaBlock code and compares it visually with common and less-common activation functions.
What you can do after
Apply ReLU through PyTorch's nn.ReLU layer inside a convolutional block.

Read how Deep Learning with PyTorch, Second Edition explains ReLU →

Data Without Labels

A rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.

How it is used
Uses ReLU in hidden layers to train neural networks faster and add nonlinearity in CNNs.
How deep it goes
Defines the piecewise function and key computational properties, while deferring other activation functions.
Examples
Uses F(x)=max(0,x), a ReLU graph, and CNN processing after convolution extracts image features.
What you can do after
Evaluate ReLU as a fast hidden-layer activation and explain its role in modeling complex CNN patterns.

Read how Data Without Labels explains ReLU →

Learn Generative AI with PyTorch

A piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.

How it is used
Uses ReLU as the default hidden-layer activation for introducing nonlinearity in PyTorch neural networks.
How deep it goes
Explains ReLU through weighted-sum decisions, nonlinear outputs, layer placement, and code, without deriving optimization behavior.
Examples
Demonstrates ReLU in an ankle-boot classifier and a two-dimensional discriminator network.
What you can do after
Apply ReLU via PyTorch's nn.ReLU in hidden layers of binary classifiers and discriminator networks.

Read how Learn Generative AI with PyTorch explains ReLU →

Timeless Algorithms, The Foundational Ideas

A rectified linear unit (ReLU) is an activation function that outputs the greater of zero and its input.

How it is used
Shapes hidden-layer representations and supports learning; deep hidden layers often use ReLU because it preserves stronger gradients for positive inputs.
How deep it goes
Defines ReLU, compares outputs with sigmoid and tanh on two inputs, and notes its positive-input gradient advantage.
Examples
Uses pre-activation inputs 1.1 and 0.1 to compare ReLU outputs with sigmoid and tanh.
What you can do after
Evaluate how ReLU changes hidden representations relative to sigmoid and tanh, and understand its stated positive-input gradient advantage.

Read how Timeless Algorithms, The Foundational Ideas explains ReLU →

Grokking AI Algorithms, Second Edition

A ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

How it is used
Uses ReLU to add nonlinear learning capacity and selectively deactivate neurons in artificial and language-model neural networks.
How deep it goes
Explains the elementwise rule and traces a vector example, but does not develop derivatives or optimization details.
Examples
Demonstrates ReLU on a five-value vector and places it after weight multiplication and bias addition in a projection layer.
What you can do after
Apply ReLU element by element after a neural-network layer’s weighted sum and bias addition.

Read how Grokking AI Algorithms, Second Edition explains ReLU →

CUDA for LLMs

ReLU is an activation function that returns the maximum of zero and its input.

How it is used
Uses ReLU as a neural-network activation and as an example of an element-wise GPU workload.
How deep it goes
Derives the forward function and backward gradient mask, then frames element-wise execution and fused-kernel design.
Examples
Illustrates ReLU on a small matrix and in the composite operation ReLU(LayerNorm(x)).
What you can do after
Build a parallelization strategy for a fused ReLU(LayerNorm(x)) kernel.

Read how CUDA for LLMs explains ReLU →

Also covered in

What to know first

Related concepts

Questions about ReLU

What is ReLU?

ReLU is an activation function that maps an input x to the greater of x and zero.

Which Manning books explain ReLU?

Grokking Machine Learning, Build AI Drug Discovery Pipelines, Inside Deep Learning: Math, Algorithms, Models, Sutskever's List, Grokking Machine Learning, Second Edition, Deep Learning with R, Third Edition, LLMs in Production, Deep Learning with Python, Third Edition and 8 more.

What should I know before learning ReLU?

Manning books that teach ReLU usually introduce classifier, language model, linear regression and objective function first.

What is ReLU a kind of?

ReLU is a kind of activation function. An activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.

What is ReLU also called?

ReLU is also called rectified linear unit or ReLUs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.