liveBook · Manning

ReLU

As explained in Grokking Machine Learning, Second Edition

A rectified linear unit (ReLU) is an activation function that returns zero for negative inputs and the input itself for nonnegative inputs.

A rectified linear unit (ReLU) is an activation function that keeps nonnegative inputs unchanged and turns negative inputs into zero: ReLU(x) = x for x ≥ 0, and ReLU(x) = 0 for x < 0. section 10.14

ReLU helps a neural network learn complex, nonlinear boundaries. Its derivative is 1 for positive inputs, making it a practical response to the vanishing-gradient problem. section 10.14 section 10.25

Apply the rule in three quick steps:

ReLU inside neural-network models

A two-dimensional classifier uses hidden layers of 128 and 64 nodes, each with ReLU activation, followed by a two-node softmax output. Trained for 100 epochs, the model reaches 90% accuracy on the dataset. A housing-price regression model similarly uses ReLU hidden layers, but ends with one output node without an activation function. section 10.18 section 10.20

Inspect the piecewise equation to see exactly how ReLU transforms negative and nonnegative values. section 10.14

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines ReLU mathematically, explains its gradient advantage, and demonstrates it in Keras architectures without deriving implementation internals.
Examples
Demonstrates ReLU with a circular two-class dataset and a Hyderabad housing-price regression model.
What it assumes
Assumes basic neural-network components such as layers and activation functions while introducing ReLU from its input-output rule.
What you can do after
Apply ReLU via Keras when configuring hidden layers for neural-network classification or regression.
How it connects
Connects ReLU to activation functions, neural networks, perceptrons, vanishing gradients, dropout, sigmoid, tanh, softmax, and backpropagation.

Also explained in

Compare how 16 books explain ReLU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.