How deeply does Deep Learning with R, Third Edition cover ReLU?

Defines ReLU, explains its tensor and geometric roles, and demonstrates implementation, while deferring deeper treatment of nonlinear transformations.

ReLU

As explained in Deep Learning with R, Third Edition

A rectified linear unit is the activation function that maps an input x to max(x, 0).

ReLU, or rectified linear unit, maps each input x to max(x, 0): positive values pass through, while negative values become zero. Applied after an affine transformation in a Dense layer, it gives neural networks nonlinear representational power. section 2.3 section 2.3.5

ReLU inside a Dense layer

A Dense layer computes an affine transformation and then applies ReLU: output = relu(dot(input, W) + b). In a sentiment classifier, two intermediate Dense layers use 16 units and ReLU before a final sigmoid output. section 4.1.3

Use ReLU with these practical ideas in mind:

Treat it as an element-wise operation: apply max(value, 0) independently to every tensor entry. section 2.3.1
Prefer optimized array operations such as pmax over nested loops; vectorized implementations are substantially faster. section 2.3.1
Remember why it matters: stacking affine layers without activations remains equivalent to a single affine transformation. section 2.3.5

Listing 4.4 Model definition

model <- keras_model_sequential() |>
  layer_dense(16, activation = "relu") |>
  layer_dense(16, activation = "relu") |>
  layer_dense(1, activation = "sigmoid")

Inspect a complete Keras model definition showing ReLU in intermediate Dense layers and sigmoid in the output layer. section 4.1.3

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Grokking Machine LearningAn activation function that returns zero for negative inputs and the input itself for nonnegative inputs.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
liveBook manning.com

Sutskever's ListReLU (rectified linear unit) is an activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
liveBook manning.com

LLMs in ProductionA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
liveBook manning.com

Deep Learning with Python, Third EditionA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
liveBook manning.com

Machine Learning Algorithms in DepthReLU is the rectified-linear-unit activation function that outputs the greater of zero and its input.
liveBook manning.com

Math and Architectures of Deep LearningThe rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
liveBook manning.com

Data Without LabelsA rectified linear unit (ReLU) is an activation function that returns its positive input and zero for nonpositive input.
liveBook manning.com

Learn Generative AI with PyTorchA piecewise linear neural-network activation function that outputs zero for negative inputs and the input value for nonnegative inputs.
liveBook manning.com

Grokking AI Algorithms, Second EditionA ReLU is an activation function that outputs zero for negative inputs and a linearly increasing value for positive inputs.

Compare how 12 books explain ReLU →

Related concepts

activation functionkind of · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.
vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
tanhcontrasts with · The hyperbolic tangent, or tanh, is a smooth nonlinear function that maps real-valued inputs to values between −1 and 1.
Swishcontrasts with · Swish, also called SiLU, is a smooth neural-network activation function defined as x multiplied by the sigmoid of βx.
SwiGLUalternative to · SwiGLU is a gated linear-unit activation variant that multiplies a SiLU-activated linear projection by a second learned linear projection.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.