liveBook · Manning

Exploding gradient

An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence.

How 6 Manning books explain it

Build AI Drug Discovery Pipelines

The exploding gradient problem is a training failure in which gradients grow excessively during backpropagation, causing unstable parameter updates.

How it is used
Introduces exploding gradients as a training-stability problem in gradient descent and deep neural networks.
How deep it goes
Explains gradient cliffs and backpropagation products, then describes clipping and initialization approaches.
Examples
Uses a gradient-cliff landscape and a SMILES autoencoder with Kaiming-initialized linear layers.
What you can do after
Evaluate and understand how gradient clipping and weight initialization address different sources of unstable training.

Read how Build AI Drug Discovery Pipelines explains exploding gradient →

Transformers in Action

The exploding gradient problem is a training failure in which gradients become excessively large, causing unstable parameter updates and potentially preventing convergence.

How it is used
Motivates attention-based architectures by contrasting their sequence connections with recurrent gradient instability.
How deep it goes
Explains recurrent gradient growth, clipping, and attention-score scaling, but does not implement gradient clipping.
Examples
Illustrates recurrent training on repetitive text or extreme numeric sequences and scaled attention with key vectors of 512 dimensions.
What you can do after
Evaluate and understand how gradient clipping and attention-score scaling address sources of training instability.

Read how Transformers in Action explains exploding gradient →

LLMs in Production

An exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.

How it is used
Explains why sequence processing makes deep recurrent neural networks difficult to train and motivates LSTM architectures.
How deep it goes
Defines the failure mode and demonstrates its sequence-length consequence, but does not detail derivative calculations or direct stabilization procedures.
Examples
Uses paired sentiment sentences and PyTorch RNN/LSTM implementations trained on Twitter sentiment data.
What you can do after
Understand how exploding gradients destabilize recurrent training and why LSTM memory cells and gates provide an alternative.

Read how LLMs in Production explains exploding gradient →

Inside Deep Learning: Math, Algorithms, Models

A training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.

How it is used
Motivates clipping as a way to limit oversized updates that disrupt neural-network training.
How deep it goes
Defines exploding gradients by their norm, then explains elementwise clipping and demonstrates a PyTorch hook.
Examples
Uses a threshold-5 gradient vector example, a PyTorch clamp hook, and contour plots of gradient-descent paths.
What you can do after
Apply gradient clipping in PyTorch by registering parameter hooks that clamp gradients to a chosen bound.

Read how Inside Deep Learning: Math, Algorithms, Models explains exploding gradient →

Evolutionary Deep Learning: Genetic algorithms and neural networks

An exploding gradient is a neural-network training failure in which backpropagated gradient values become extremely large across successive layers.

How it is used
Diagnoses unstable backpropagation in deeper or more complex neural networks.
How deep it goes
Explains signal magnification across layers and normalization-based mitigation, but stops short of detailed optimization algorithms.
Examples
Uses a deep-network problem figure and a convolutional feature-map calculation to illustrate gradient instability and normalization.
What you can do after
Diagnose gradient instability and evaluate normalization or dropout as practical architectural responses.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains exploding gradient →

Machine Learning System Design

An exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.

How it is used
Diagnoses unstable training from loss curves and guides practical checks before deployment.
How deep it goes
Identifies symptoms and provides stabilization checks, but stops short of mathematical derivation or algorithmic detail.
Examples
Uses oscillating, exploding, or NaN-valued loss curves and small-batch overfitting as debugging artifacts.
What you can do after
Diagnose unstable training and apply learning-rate, clipping, initialization, data, and model-complexity checks.

Read how Machine Learning System Design explains exploding gradient →

Also covered in

What to know first

Related concepts

Questions about exploding gradient

What is exploding gradient?

An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence.

Which Manning books explain exploding gradient?

Build AI Drug Discovery Pipelines, Transformers in Action, LLMs in Production, Inside Deep Learning: Math, Algorithms, Models, Evolutionary Deep Learning: Genetic algorithms and neural networks and Machine Learning System Design.

What should I know before learning exploding gradient?

Manning books that teach exploding gradient usually introduce gradient, objective function, parameter and backpropagation first.

What is the difference between exploding gradient and vanishing gradient problem?

An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence. The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.