liveBook · Manning

Unstable gradient problem

As explained in Build AI Drug Discovery Pipelines

The unstable gradient problem is the failure of gradients in deep networks to maintain a useful scale as they propagate across layers.

The unstable gradient problem occurs when gradients lose a useful scale during backpropagation. Products above 1 produce exploding gradients; products below 1 produce vanishing gradients, so early layers learn at dramatically different speeds or not at all. section 8.2.2

Common responses include:

  • Use Xavier or He initialization to adjust random weights according to the previous layer’s size. section 8.2.2
  • Use ReLU to help keep gradient updates stable during training. section 8.2.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains gradient products across layers and gives initialization and activation remedies without developing a formal derivation.
Examples
Uses a kinase-binder network, ReLU comparisons, and small random parameter values as concrete illustrations.
What you can do after
Apply it via PyTorch by choosing stable initialization, ReLU activations, and an appropriately scaled network architecture.
How it connects
Connects backpropagation with exploding gradients, vanishing gradients, weight initialization, activation functions, and ReLU.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.