liveBook · Manning

Exploding gradient

As explained in Build AI Drug Discovery Pipelines

The exploding gradient problem is a training failure in which gradients grow excessively during backpropagation, causing unstable parameter updates.

An exploding gradient occurs when gradients become excessively large during training. Because gradient descent uses gradients to update model weights, a large gradient can push the weights too far across the cost-function landscape and make training unstable. section 4.2.3 section 10.2.3

One cause is a gradient cliff: a steep drop in the cost function produces a sudden increase in the gradient. In a deep neural network, gradients can also grow during backpropagation. The gradient for an early layer includes a product of terms from later layers; if those terms are greater than one, the product grows as more layers are traversed. Large initial weights can contribute to this problem. Weights that are too small can instead produce vanishing gradients. section 4.2.3 section 8.2.2 section 10.2.3

Figure 4.13 Gradient cliffs impact converging to an optimal minimum of model weights.

The gradient-cliff illustration shows how a steep region can disrupt convergence toward optimal weights. section 4.2.3

Two ways to address the problem act at different points in training:

  • Gradient clipping limits gradients during training. One approach sets minimum and maximum values that a gradient cannot cross. section 4.2.3
  • Weight initialization sets an appropriate starting scale. Xavier and He initialization adjust random weights according to layer size to help maintain gradient scale through a network. section 8.2.2

Initializing a molecular autoencoder

An autoencoder that compresses and reconstructs molecular SMILES strings uses Kaiming (He) initialization for its ReLU-based linear layers. With fan-out mode and 256 outputs, the initial weights are sampled uniformly from approximately −0.153 to 0.153. Biases start at zero; embedding weights receive a separate small normal initialization. section 10.2.3

Listing 10.6 Kaiming Initialization
   def _initialize_weights(self):
        for module in self.modules():
            if isinstance(module, nn.Linear):    #A
                nn.init.kaiming_uniform_(
                    module.weight, mode='fan_out', nonlinearity='relu'
                )
                if module.bias is not None:    #B
                    nn.init.zeros_(module.bias)
            elif isinstance(module, nn.Embedding):
                nn.init.normal_(module.weight, mean=0, std=0.1)
                if module.padding_idx is not None:
                    module.weight.data[module.padding_idx].zero_()

The PyTorch initialization routine applies Kaiming initialization to linear weights and separately initializes biases and embeddings. section 10.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains gradient cliffs and backpropagation products, then describes clipping and initialization approaches.
Examples
Uses a gradient-cliff landscape and a SMILES autoencoder with Kaiming-initialized linear layers.
What you can do after
Evaluate and understand how gradient clipping and weight initialization address different sources of unstable training.
How it connects
Connects gradient descent and backpropagation with gradient clipping, weight initialization, and vanishing gradients.

Also explained in

Compare how 6 books explain exploding gradient →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.