liveBook · Manning

Gradient cliff

As explained in Build AI Drug Discovery Pipelines

A gradient cliff is a region of an objective landscape where the cost changes steeply over a small parameter displacement.

A gradient cliff is a region of the cost landscape where a small parameter displacement produces a steep cost change. During gradient descent, the resulting large gradient can move model weights too far and destabilize convergence. section 4.2.3

Use gradient clipping to limit the update caused by a cliff:

  • Set minimum and maximum clip values so gradients cannot fall below or exceed chosen bounds. section 4.2.3
  • Treat clipping as a practical response to exploding gradients that make optimization jump across the cost landscape. section 4.2.3
Figure 4.13 Gradient cliffs impact converging to an optimal minimum of model weights.

Visualizes how gradient cliffs interfere with reaching an optimal minimum of the model weights. section 4.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains the landscape effect and clipping response, but stops short of implementation code or clipping-parameter selection.
Examples
Uses a cost-landscape figure showing gradient cliffs disrupting convergence toward optimal model weights.
What it assumes
Assumes basic gradient descent and cost-landscape terminology while introducing the cliff and exploding-gradient failure mode.
What you can do after
Diagnose exploding-gradient behavior and apply bounded gradient clipping as a stabilization strategy.
How it connects
Connects gradient descent with exploding gradients, gradient clipping, cost functions, and feature scaling.

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.