liveBook · Manning

Gradient clipping

As explained in Build AI Drug Discovery Pipelines

Gradient clipping is a training technique that limits gradient values or their norm to prevent excessively large parameter updates.

Gradient clipping limits gradient values during gradient descent so an exploding gradient does not push model weights too far across the cost-function landscape. It is useful when training models for solubility prediction, where unstable updates can disrupt convergence. section 4.2.3

Apply the technique by:

  • Identifying a gradient cliff: a steep drop in the cost function that produces an unusually large gradient. section 4.2.3
  • Setting minimum and maximum clip values, then preventing each gradient from falling below or exceeding those bounds. section 4.2.3
  • Considering learning rate, convergence criteria, and feature scaling alongside clipping when diagnosing gradient-descent behavior. section 4.5 section 4.2.3
Figure 4.13 Gradient cliffs impact converging to an optimal minimum of model weights.

Use the figure to see how gradient cliffs interfere with reaching an optimal minimum of the model weights. section 4.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines value clipping and its motivation, but stops short of implementation code or norm-based clipping.
Examples
Uses a gradient-cliff figure and solubility-model training as concrete contexts.
What it assumes
Assumes gradient descent and cost functions, while connecting clipping to exploding gradients and convergence.
What you can do after
Diagnose exploding-gradient behavior and apply minimum and maximum bounds to gradient values.
How it connects
Links gradient descent with exploding gradients, learning rate, convergence criteria, and feature scaling.

Also explained in

Compare how 3 books explain gradient clipping →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.