Exploding gradient

As explained in Machine Learning System Design

An exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.

An exploding gradient appears when gradient magnitudes become excessively large, causing loss to explode or trend toward NaN and making training unstable. Treat it as a signal to inspect both optimization settings and implementation or preprocessing errors. section 9.1.3

Use this troubleshooting sequence when the loss suddenly rises or becomes NaN:

• Check feature–target correlation, sample–label ordering, and possible NaNs in the data. section 9.1.3
• Lower the learning rate to reduce unstable movement through parameter space. section 9.1.3
• Try gradient clipping or different weight-initialization techniques when the gradient itself has exploded. section 9.1.3
• Test a single batch or 10–100 samples, overfit them, and add model complexity only after a simpler baseline works. section 9.1.3

Figure 9.4 A model was converging until something went wrong.

Use the loss-curve figure to recognize training that converges before something goes wrong. section 9.1.3

How Machine Learning System Design covers it

How deep it goes

Identifies symptoms and provides stabilization checks, but stops short of mathematical derivation or algorithmic detail.

Examples

Uses oscillating, exploding, or NaN-valued loss curves and small-batch overfitting as debugging artifacts.

What you can do after

Diagnose unstable training and apply learning-rate, clipping, initialization, data, and model-complexity checks.

How it connects

Connects exploding gradients with loss curves, gradient clipping, learning rate, weight initialization, NaNs, and data preprocessing.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionThe exploding gradient problem is a training failure in which gradients become excessively large, causing unstable parameter updates and potentially preventing convergence.
• 

  liveBookmanning.com
  LLMs in ProductionAn exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksAn exploding gradient is a neural-network training failure in which backpropagated gradient values become extremely large across successive layers.

Compare how 5 books explain exploding gradient →

Related concepts

• gradient clippingaddressed by · Gradient clipping is an optimization technique that limits gradient values or their norm before parameter updates to reduce excessively large or unstable updates.
• normalizationaddressed by · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
• weight initializationaddressed by · Weight initialization is the process of assigning starting values to a model’s trainable parameters before training or optimization.
• batch normalizationaddressed by · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
• recurrent neural networkproduced by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.