An exploding gradient occurs when gradients become excessively large during training. Because gradient descent uses gradients to update model weights, a large gradient can push the weights too far across the cost-function landscape and make training unstable. section 4.2.3 section 10.2.3
One cause is a gradient cliff: a steep drop in the cost function produces a sudden increase in the gradient. In a deep neural network, gradients can also grow during backpropagation. The gradient for an early layer includes a product of terms from later layers; if those terms are greater than one, the product grows as more layers are traversed. Large initial weights can contribute to this problem. Weights that are too small can instead produce vanishing gradients. section 4.2.3 section 8.2.2 section 10.2.3
Figure 4.13 Gradient cliffs impact converging to an optimal minimum of model weights.

The gradient-cliff illustration shows how a steep region can disrupt convergence toward optimal weights. section 4.2.3
Two ways to address the problem act at different points in training:
- Gradient clipping limits gradients during training. One approach sets minimum and maximum values that a gradient cannot cross. section 4.2.3
- Weight initialization sets an appropriate starting scale. Xavier and He initialization adjust random weights according to layer size to help maintain gradient scale through a network. section 8.2.2
Initializing a molecular autoencoder
An autoencoder that compresses and reconstructs molecular SMILES strings uses Kaiming (He) initialization for its ReLU-based linear layers. With fan-out mode and 256 outputs, the initial weights are sampled uniformly from approximately −0.153 to 0.153. Biases start at zero; embedding weights receive a separate small normal initialization. section 10.2.3
Listing 10.6 Kaiming Initialization
def _initialize_weights(self):
for module in self.modules():
if isinstance(module, nn.Linear): #A
nn.init.kaiming_uniform_(
module.weight, mode='fan_out', nonlinearity='relu'
)
if module.bias is not None: #B
nn.init.zeros_(module.bias)
elif isinstance(module, nn.Embedding):
nn.init.normal_(module.weight, mean=0, std=0.1)
if module.padding_idx is not None:
module.weight.data[module.padding_idx].zero_()The PyTorch initialization routine applies Kaiming initialization to linear weights and separately initializes biases and embeddings. section 10.2.3





