LeakyReLU modifies ReLU so negative inputs retain a small slope instead of becoming a hard zero. For leak factor α, it returns x when x > 0 and αx when x ≤ 0; its derivative is 1 on the positive side and α on the negative side. This preserves a nonzero local gradient and can prevent neurons from becoming permanently inactive, especially in small networks. section 6.1.2
Use the activation with these practical considerations:
- Choose a small leak such as α = 0.01–0.3; α = 0.1 is a preferred general default, while PyTorch defaults to 0.01. section 6.1.2 section 6.1.3
- Expect ReLU and LeakyReLU to behave linearly for positive inputs; on the negative side, LeakyReLU continues decreasing rather than remaining fixed at zero. section 6.1.2
- Remember that LeakyReLU prevents its own gradient from vanishing, but interactions between layers can still create exploding or vanishing gradients. section 6.1.2
- Use larger leaks such as α = 0.1 or 0.2 in GANs, where discriminator gradients must pass through to train the generator. section 9.1.3
Replacing activations in PyTorch models
A fully connected model can place nn.LeakyReLU(leak_rate) after each linear layer. A CNN can centralize Conv2d plus LeakyReLU in a reusable cnnLayer function, allowing the activation to change without rewriting the architecture. Training comparable fully connected and convolutional models commonly produces better accuracy and numerically cleaner behavior than tanh. section 6.1.3
def leaky_relu(x, alpha=0.1): #1
return max(alpha*x, x) def leaky_reluP(x, alpha=0.1): #2
if x > 0:
return 1
else:
return alpha
Inspect the minimal forward function and derivative implementation for LeakyReLU. section 6.1.2
def simpleGAN(latent_d, neurons, out_shape, sigmoidG=False, leak=0.2):
"""
This function will create a simple GAN for us to train. It will return
a tuple (G, D), holding the generator and discriminator network
respectively.
latent_d: the number of latent variables we will use as input to the
generator G.
neurons: how many hidden neurons to use in each hidden layer
out_shape: the shape of the output of the discriminator D. This should
be the same shape as the real data.
sigmoidG: true if the generator G should end with a sigmoid activation,
or False if it should just return unbounded activations
"""
G = nn.Sequential( fcLayer(latent_d, neurons, leak),
fcLayer(neurons, neurons, leak),
fcLayer(neurons, neurons, leak),
nn.Linear(neurons, abs(np.prod(out_shape)) ), #1
View(out_shape) #2
)
if sigmoidG: #3
G = nn.Sequential(G,
nn.Sigmoid())
D = nn.Sequential( nn.Flatten(),
fcLayer(abs(np.prod(out_shape)), neurons, leak),
fcLayer(neurons, neurons, leak),
fcLayer(neurons, neurons, leak),
nn.Linear(neurons, 1 ) #4
)
return G, D
#1 np.prod multiplies each value in the shape, giving us the total number of needed outputs. abs removes the impact of “–1" for the batch dimension.
#2 Reshapes the output to whatever D expects
#3 Sometimes we do or don’t want G to return a sigmoid value (i.e., [0,1]), so we rap it in a conditional.
#4 D has one output for a binary classification problem.
See LeakyReLU embedded in reusable GAN generator and discriminator layers. section 9.1.3
