Exploding gradient
An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence.
Build AI Drug Discovery Pipelines
The exploding gradient problem is a training failure in which gradients grow excessively during backpropagation, causing unstable parameter updates.
- How it is used
- Introduces exploding gradients as a training-stability problem in gradient descent and deep neural networks.
- How deep it goes
- Explains gradient cliffs and backpropagation products, then describes clipping and initialization approaches.
- Examples
- Uses a gradient-cliff landscape and a SMILES autoencoder with Kaiming-initialized linear layers.
- What you can do after
- Evaluate and understand how gradient clipping and weight initialization address different sources of unstable training.
Read how Build AI Drug Discovery Pipelines explains exploding gradient →
Transformers in Action
The exploding gradient problem is a training failure in which gradients become excessively large, causing unstable parameter updates and potentially preventing convergence.
- How it is used
- Motivates attention-based architectures by contrasting their sequence connections with recurrent gradient instability.
- How deep it goes
- Explains recurrent gradient growth, clipping, and attention-score scaling, but does not implement gradient clipping.
- Examples
- Illustrates recurrent training on repetitive text or extreme numeric sequences and scaled attention with key vectors of 512 dimensions.
- What you can do after
- Evaluate and understand how gradient clipping and attention-score scaling address sources of training instability.
Read how Transformers in Action explains exploding gradient →
LLMs in Production
An exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.
- How it is used
- Explains why sequence processing makes deep recurrent neural networks difficult to train and motivates LSTM architectures.
- How deep it goes
- Defines the failure mode and demonstrates its sequence-length consequence, but does not detail derivative calculations or direct stabilization procedures.
- Examples
- Uses paired sentiment sentences and PyTorch RNN/LSTM implementations trained on Twitter sentiment data.
- What you can do after
- Understand how exploding gradients destabilize recurrent training and why LSTM memory cells and gates provide an alternative.
Inside Deep Learning: Math, Algorithms, Models
A training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.
- How it is used
- Motivates clipping as a way to limit oversized updates that disrupt neural-network training.
- How deep it goes
- Defines exploding gradients by their norm, then explains elementwise clipping and demonstrates a PyTorch hook.
- Examples
- Uses a threshold-5 gradient vector example, a PyTorch clamp hook, and contour plots of gradient-descent paths.
- What you can do after
- Apply gradient clipping in PyTorch by registering parameter hooks that clamp gradients to a chosen bound.
Read how Inside Deep Learning: Math, Algorithms, Models explains exploding gradient →
Evolutionary Deep Learning: Genetic algorithms and neural networks
An exploding gradient is a neural-network training failure in which backpropagated gradient values become extremely large across successive layers.
- How it is used
- Diagnoses unstable backpropagation in deeper or more complex neural networks.
- How deep it goes
- Explains signal magnification across layers and normalization-based mitigation, but stops short of detailed optimization algorithms.
- Examples
- Uses a deep-network problem figure and a convolutional feature-map calculation to illustrate gradient instability and normalization.
- What you can do after
- Diagnose gradient instability and evaluate normalization or dropout as practical architectural responses.
Machine Learning System Design
An exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.
- How it is used
- Diagnoses unstable training from loss curves and guides practical checks before deployment.
- How deep it goes
- Identifies symptoms and provides stabilization checks, but stops short of mathematical derivation or algorithmic detail.
- Examples
- Uses oscillating, exploding, or NaN-valued loss curves and small-batch overfitting as debugging artifacts.
- What you can do after
- Diagnose unstable training and apply learning-rate, clipping, initialization, data, and model-complexity checks.
Read how Machine Learning System Design explains exploding gradient →
Also covered in
- Machines that ThinkIn early access (MEAP)
- Learn Generative AI with PyTorch
- Deep Learning with R, Third Edition
- Deep Learning with Python, Third Edition
- Grokking AI Algorithms, Second Edition
- CUDA for LLMs
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Related concepts
- vanishing gradient problemcontrasts with · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.
- gradient clippingaddressed by · Gradient clipping is an optimization technique that limits gradient values or their norm before parameter updates to reduce excessively large or unstable updates.
- clippingaddressed by · Clipping is the operation of constraining values or content to specified bounds or regions by limiting or removing what falls outside them.
- normalizationaddressed by · Normalization is the process of transforming or organizing data or representations according to rules to achieve a consistent form, scale, or structure.
- weight initializationaddressed by · Weight initialization is the process of assigning initial values to a model’s trainable parameters before training or optimization.
- batch normalizationaddressed by · Batch normalization is a neural-network technique that normalizes activations using statistics computed across a batch during training and stored running statistics during inference.
Questions about exploding gradient
What is exploding gradient?
An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence.
Which Manning books explain exploding gradient?
Build AI Drug Discovery Pipelines, Transformers in Action, LLMs in Production, Inside Deep Learning: Math, Algorithms, Models, Evolutionary Deep Learning: Genetic algorithms and neural networks and Machine Learning System Design.
What should I know before learning exploding gradient?
Manning books that teach exploding gradient usually introduce gradient, objective function, parameter and backpropagation first.
What is the difference between exploding gradient and vanishing gradient problem?
An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence. The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





