Exploding gradient

As explained in Inside Deep Learning: Math, Algorithms, Models

A training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.

An exploding gradient occurs when the gradient magnitude becomes excessively large, producing parameter updates that overshoot the intended optimization path. With ||g_t||₂ approaching infinity, even a small learning rate can cause performance degradation or prevent learning; very deep networks are one reported cause. section 5.1.2

Clipping an extreme gradient

With threshold z = 5, g = [1, −2, 1000, 3, −7] becomes [1, −2, 5, 3, −5]. Clipping preserves the indicated direction while limiting the unreasonable step size. section 5.3.3

Gradient clipping addresses the problem by clamping every gradient value to [−5, 5]. In PyTorch, register_hook can apply torch.clamp to each parameter gradient before the optimizer uses it. The technique works with Adam, SGD, and learning-rate schedules, but may worsen results when exploding gradients are absent. section 5.3.3

Figure 5.2 A toy example showing three scenarios for gradient descent. Each is a contour plot of a 2D optimization going from red (high values, bad) to dark green (low values, good). The ideal case is at the top left, where each gradient heads exactly toward the solution. The top right shows noisy gradients that cause the descent to head in not quite the correct direction, requiring more steps. The bottom left shows exploding gradients, where we begin to take steps that are far too large and head away from the solution. The bottom right shows vanishing gradients, where the gradient becomes so small that we have to take an excessive number of steps to get to the solution.

Compare ideal, noisy, exploding, and vanishing gradient-descent paths in a two-dimensional contour plot. section 5.1.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Defines the instability mathematically, illustrates its effect, and demonstrates value-wise clipping without deriving gradient internals.

Examples

Uses a clipped five-element gradient, a contour plot, and PyTorch parameter hooks with torch.clamp.

What it assumes

Assumes gradient descent, gradients, learning rates, model parameters, and optimizers; contrasts exploding gradients with vanishing gradients.

What you can do after

Diagnose unstable training and apply PyTorch gradient clipping through parameter hooks.

How it connects

Connects exploding gradients to gradient descent, learning rate, vanishing gradients, gradient clipping, optimizers, and recurrent neural networks.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionThe exploding gradient problem is a training failure in which gradients become excessively large, causing unstable parameter updates and potentially preventing convergence.
• 

  liveBookmanning.com
  LLMs in ProductionAn exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksAn exploding gradient is a neural-network training failure in which backpropagated gradient values become extremely large across successive layers.
• 

  liveBookmanning.com
  Machine Learning System DesignAn exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.

Compare how 5 books explain exploding gradient →

Related concepts

• gradient clippingaddressed by · Gradient clipping is an optimization technique that limits gradient values or their norm before parameter updates to reduce excessively large or unstable updates.
• normalizationaddressed by · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
• weight initializationaddressed by · Weight initialization is the process of assigning starting values to a model’s trainable parameters before training or optimization.
• batch normalizationaddressed by · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
• recurrent neural networkproduced by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.