liveBook · Manning

Vanishing gradient problem

As explained in Build AI Drug Discovery Pipelines

A vanishing gradient is a training failure mode in which gradients become extremely small during backpropagation, preventing earlier layers from learning effectively.

The vanishing gradient problem occurs when the learning signal becomes extremely small during backpropagation, the process used to determine how to adjust a network’s weights. Early layers then receive tiny updates and may learn very slowly or not at all. section 8.2.2 section 10.2.3

As a gradient travels backward through layers, it is multiplied by terms from later layers. Repeated multiplication by values below one can make the signal shrink toward zero. The opposite failure, exploding gradients, occurs when the signal grows too large. Both can make deeper networks difficult to train. section 8.2.2

Long molecular sequences

A recurrent neural network (RNN) backpropagates through the steps of a sequence. Over many steps, its gradients can become too weak to learn relationships between distant tokens. This matters when generating SMILES, a way to represent molecules as strings: the model must match ring markers that may be far apart, as in the two “1” markers in c1ccccc1 for benzene. section 10.4.2

Different designs address sources of weak gradients:

  • Weight initialization: Xavier or He initialization adjusts the scale of random weights; Kaiming (He) initialization is designed for networks using ReLU activations. section 8.2.2 section 10.2.3
  • Gated recurrent units (GRUs): reset and update gates control how previous information and new information contribute to the hidden state, helping capture long-range sequence dependencies. section 10.4.2
  • Scaled dot-product attention: dividing attention scores by the square root of the key dimension helps prevent large scores from pushing softmax into regions with tiny gradients. section 12.3.2
Listing 10.6 Kaiming Initialization
   def _initialize_weights(self):
        for module in self.modules():
            if isinstance(module, nn.Linear):    #A
                nn.init.kaiming_uniform_(
                    module.weight, mode='fan_out', nonlinearity='relu'
                )
                if module.bias is not None:    #B
                    nn.init.zeros_(module.bias)
            elif isinstance(module, nn.Embedding):
                nn.init.normal_(module.weight, mean=0, std=0.1)
                if module.padding_idx is not None:
                    module.weight.data[module.padding_idx].zero_()

This PyTorch initialization routine applies Kaiming initialization to linear weights in a SMILES autoencoder, with separate handling for biases and embeddings. section 10.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains gradient shrinkage through layers and timesteps, then describes initialization, gating, and attention scaling as responses.
Examples
Uses SMILES ring closures, a kinase-binding network, and a molecular autoencoder to illustrate training challenges and design choices.
What you can do after
Evaluate and understand how weak gradients can impair learning and why specific network design choices address them.
How it connects
Connects backpropagation with weight initialization, ReLU, gated recurrent units, and scaled dot-product attention.

Also explained in

Compare how 17 books explain vanishing gradient problem →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.