ReLU, or rectified linear unit, is a piecewise linear activation function: it outputs zero for negative inputs and the input value for nonnegative inputs. It helps neural networks learn nonlinear relationships by deciding whether a neuron turns on from its weighted sum. section 2.3.2 section 2.5
A common pattern uses ReLU in hidden layers and sigmoid at the final layer. ReLU supplies hidden-layer nonlinearity, while sigmoid converts the final value into a probability for binary classification. section 2.3.2
A practical PyTorch network applies ReLU between dense layers:
- Flatten each 28 × 28 image into 784 values before the first dense layer. section 2.3.2
- Pass activations through dense layers with 256, 128, and 32 neurons, applying ReLU after each hidden layer. section 2.3.2
- Finish with one output, dropout, and sigmoid for ankle-boot versus t-shirt prediction. section 2.3.2
ReLU in a discriminator
A PyTorch discriminator can use ReLU after each of three dense layers, with dropout between layers, then produce one sigmoid output representing the probability that a sample is real. section 3.3.1
Listing 2.5 Creating a binary classification model
import torch.nn as nn
device="cuda" if torch.cuda.is_available() else "cpu" #1
binary_model=nn.Sequential( #2
nn.Linear(28*28,256), #3
nn.ReLU(), #4
nn.Linear(256,128),
nn.ReLU(),
nn.Linear(128,32),
nn.ReLU(),
nn.Linear(32,1),
nn.Dropout(p=0.25),
nn.Sigmoid()).to(device) #5
Inspect the complete sequential binary-classification model, including three nn.ReLU layers and the final sigmoid. section 2.3.2
