liveBook · Manning

ReLU

As explained in Timeless Algorithms, The Foundational Ideas

A rectified linear unit (ReLU) is an activation function that outputs the greater of zero and its input.

ReLU sends negative inputs to zero and passes positive inputs through unchanged. Its effect on a network depends on the values entering the activation: changing the activation changes hidden representations and gradients. section 11.4.1 section 11.5.3

ReLU⁡(z)=max⁡(0,z)

The equation states the ReLU rule: output the greater of zero and the input. section 11.4.1

Applying ReLU

For inputs 1.1 and 0.1, ReLU outputs 1.1 and 0.1. Unlike sigmoid and tanh, it leaves these positive values unchanged. section 11.4.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines ReLU, compares outputs with sigmoid and tanh on two inputs, and notes its positive-input gradient advantage.
Examples
Uses pre-activation inputs 1.1 and 0.1 to compare ReLU outputs with sigmoid and tanh.
What you can do after
Evaluate how ReLU changes hidden representations relative to sigmoid and tanh, and understand its stated positive-input gradient advantage.
How it connects
Connects activation functions with hidden-layer representations, gradients, sigmoid, and tanh.

Also explained in

Compare how 16 books explain ReLU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.