The vanishing gradient problem occurs when the learning signal becomes extremely small during backpropagation, the process used to determine how to adjust a network’s weights. Early layers then receive tiny updates and may learn very slowly or not at all. section 8.2.2 section 10.2.3
As a gradient travels backward through layers, it is multiplied by terms from later layers. Repeated multiplication by values below one can make the signal shrink toward zero. The opposite failure, exploding gradients, occurs when the signal grows too large. Both can make deeper networks difficult to train. section 8.2.2
Long molecular sequences
A recurrent neural network (RNN) backpropagates through the steps of a sequence. Over many steps, its gradients can become too weak to learn relationships between distant tokens. This matters when generating SMILES, a way to represent molecules as strings: the model must match ring markers that may be far apart, as in the two “1” markers in c1ccccc1 for benzene. section 10.4.2
Different designs address sources of weak gradients:
- Weight initialization: Xavier or He initialization adjusts the scale of random weights; Kaiming (He) initialization is designed for networks using ReLU activations. section 8.2.2 section 10.2.3
- Gated recurrent units (GRUs): reset and update gates control how previous information and new information contribute to the hidden state, helping capture long-range sequence dependencies. section 10.4.2
- Scaled dot-product attention: dividing attention scores by the square root of the key dimension helps prevent large scores from pushing softmax into regions with tiny gradients. section 12.3.2
Listing 10.6 Kaiming Initialization
def _initialize_weights(self):
for module in self.modules():
if isinstance(module, nn.Linear): #A
nn.init.kaiming_uniform_(
module.weight, mode='fan_out', nonlinearity='relu'
)
if module.bias is not None: #B
nn.init.zeros_(module.bias)
elif isinstance(module, nn.Embedding):
nn.init.normal_(module.weight, mean=0, std=0.1)
if module.padding_idx is not None:
module.weight.data[module.padding_idx].zero_()This PyTorch initialization routine applies Kaiming initialization to linear weights in a SMILES autoencoder, with separate handling for biases and embeddings. section 10.2.3
















