Exploding gradient
An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or numerical and convergence failures.
Transformers in Action
The exploding gradient problem is a training failure in which gradients become excessively large, causing unstable parameter updates and potentially preventing convergence.
- How it is used
- Motivates gradient clipping in recurrent models and score scaling in transformer attention as responses to unstable training.
- How deep it goes
- Explains recurrent gradient growth and attention-score scaling, but gives no clipping threshold or worked training diagnosis.
- Examples
- Uses long repetitive sequences, extreme-valued numeric sequences, a scaled-versus-unscaled variance figure, and a simplified attention listing.
- What you can do after
- Diagnose/debug with it by recognizing unstable gradient-driven updates and identifying clipping or attention-score scaling as relevant mitigations.
Read how Transformers in Action explains exploding gradient →
LLMs in Production
An exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.
- How it is used
- Explains why sequence processing makes deep recurrent neural networks difficult to train and motivates LSTM architectures.
- How deep it goes
- Defines the failure mode and demonstrates its sequence-length consequence, but does not detail derivative calculations or direct stabilization procedures.
- Examples
- Uses paired sentiment sentences and PyTorch RNN/LSTM implementations trained on Twitter sentiment data.
- What you can do after
- Understand how exploding gradients destabilize recurrent training and why LSTM memory cells and gates provide an alternative.
Inside Deep Learning: Math, Algorithms, Models
A training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.
- How it is used
- Diagnoses unstable optimization and motivates modifying gradients before parameter updates.
- How deep it goes
- Defines the instability mathematically, illustrates its effect, and demonstrates value-wise clipping without deriving gradient internals.
- Examples
- Uses a clipped five-element gradient, a contour plot, and PyTorch parameter hooks with torch.clamp.
- What you can do after
- Diagnose unstable training and apply PyTorch gradient clipping through parameter hooks.
Read how Inside Deep Learning: Math, Algorithms, Models explains exploding gradient →
Evolutionary Deep Learning: Genetic algorithms and neural networks
An exploding gradient is a neural-network training failure in which backpropagated gradient values become extremely large across successive layers.
- How it is used
- Diagnoses unstable backpropagation in deeper or more complex neural networks.
- How deep it goes
- Explains signal magnification across layers and normalization-based mitigation, but stops short of detailed optimization algorithms.
- Examples
- Uses a deep-network problem figure and a convolutional feature-map calculation to illustrate gradient instability and normalization.
- What you can do after
- Diagnose gradient instability and evaluate normalization or dropout as practical architectural responses.
Machine Learning System Design
An exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.
- How it is used
- Diagnoses unstable training from loss curves and guides practical checks before deployment.
- How deep it goes
- Identifies symptoms and provides stabilization checks, but stops short of mathematical derivation or algorithmic detail.
- Examples
- Uses oscillating, exploding, or NaN-valued loss curves and small-batch overfitting as debugging artifacts.
- What you can do after
- Diagnose unstable training and apply learning-rate, clipping, initialization, data, and model-complexity checks.
Read how Machine Learning System Design explains exploding gradient →
Related concepts
- gradient clippingaddressed by · Gradient clipping is an optimization technique that limits gradient values or their norm before parameter updates to reduce excessively large or unstable updates.
- normalizationaddressed by · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
- weight initializationaddressed by · Weight initialization is the process of assigning starting values to a model’s trainable parameters before training or optimization.
- batch normalizationaddressed by · Batch normalization is a neural-network technique that normalizes activations using statistics computed over a mini-batch, with learned scale and shift parameters and typically running statistics for inference.
- recurrent neural networkproduced by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.