A gradient cliff is a region of the cost landscape where a small parameter displacement produces a steep cost change. During gradient descent, the resulting large gradient can move model weights too far and destabilize convergence. section 4.2.3
Use gradient clipping to limit the update caused by a cliff:
- Set minimum and maximum clip values so gradients cannot fall below or exceed chosen bounds. section 4.2.3
- Treat clipping as a practical response to exploding gradients that make optimization jump across the cost landscape. section 4.2.3
Figure 4.13 Gradient cliffs impact converging to an optimal minimum of model weights.

Visualizes how gradient cliffs interfere with reaching an optimal minimum of the model weights. section 4.2.3
