ReLU sends negative inputs to zero and passes positive inputs through unchanged. Its effect on a network depends on the values entering the activation: changing the activation changes hidden representations and gradients. section 11.4.1 section 11.5.3
The equation states the ReLU rule: output the greater of zero and the input. section 11.4.1
Applying ReLU
For inputs 1.1 and 0.1, ReLU outputs 1.1 and 0.1. Unlike sigmoid and tanh, it leaves these positive values unchanged. section 11.4.1















