Vanishing gradient problem

As explained in Inside Deep Learning: Math, Algorithms, Models

A training instability in which gradients become too small for effective learning updates.

The vanishing gradient problem occurs when the gradient with respect to a network’s parameters becomes so small that weight updates make almost no progress. Formally, when ||∇ΘfΘ(x)|| approaches zero, learning effectively stops, even if the learning rate is large. section 6.1.1

Saturating activations are a common cause. For large positive inputs, tanh and sigmoid flatten near 1; for large negative inputs, they flatten near −1 and 0. Their derivatives consequently approach zero, so saturated neurons transmit little learning signal backward. section 6.1.1

Use these distinctions when diagnosing or reducing the problem:

• Distinguish vanishing gradients from exploding gradients: tiny gradients require excessive optimization steps, while huge gradients cause overly large updates. section 5.1.2
• Inspect gradient histograms when learning stalls; saturation is common but is not the only possible cause. section 6.1.1
• Try ReLU instead of tanh or sigmoid when saturation is not intentional; ReLU commonly converges more accurately in fewer epochs. section 6.1.2
• Use leaky ReLU when inactive ReLU units are costly: its negative side retains a small slope controlled by α. section 6.1.2

Activation derivatives as a quick check

For tanh, use tanh′(x) = 1 − tanh(x)²; for sigmoid, use σ′(x) = σ(x)(1 − σ(x)). Plot these derivatives over an input range such as −5 to 5: values near zero identify saturated regions. ReLU instead has derivative 1 for positive inputs and 0 for nonpositive inputs; leaky ReLU replaces the zero branch with α.

Figure 5.2 A toy example showing three scenarios for gradient descent. Each is a contour plot of a 2D optimization going from red (high values, bad) to dark green (low values, good). The ideal case is at the top left, where each gradient heads exactly toward the solution. The top right shows noisy gradients that cause the descent to head in not quite the correct direction, requiring more steps. The bottom left shows exploding gradients, where we begin to take steps that are far too large and head away from the solution. The bottom right shows vanishing gradients, where the gradient becomes so small that we have to take an excessive number of steps to get to the solution.

Compare ideal, noisy, exploding, and vanishing optimization paths in a contour plot; the vanishing case requires excessive steps because each gradient is too small. section 5.1.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains parameter-gradient updates, activation saturation, derivative plots, and ReLU variants, but does not enumerate every cause.

Examples

Uses contour plots, tanh and sigmoid derivative plots, gradient histograms, and ReLU or leaky-ReLU code.

What it assumes

Assumes gradients, parameters, loss functions, and gradient descent; teaches saturation and derivative behavior explicitly.

What you can do after

Diagnose vanishing gradients with derivative plots or gradient histograms and reduce them by selecting ReLU-family activations.

How it connects

Connects gradient descent, learning rates, momentum, activation functions, ReLU, leaky ReLU, and GAN training.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListA vanishing gradient is an optimization failure in which derivatives become too small to support effective parameter updates during neural-network training.
• 

  liveBookmanning.com
  Grokking Machine LearningA neural-network training problem in which gradients become extremely small as they are propagated through layers, slowing or preventing learning.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe training difficulty in deep networks where repeated multiplication of small derivatives makes early-layer gradients nearly zero.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA vanishing gradient is a training condition in which gradients become extremely small as they propagate through a deep network, slowing or preventing parameter updates in earlier layers.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training problem in deep networks where gradient signals become too small to effectively update earlier layers.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn optimization problem in which gradients become progressively smaller as they are propagated through many computational steps, hindering learning of long-range dependencies.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA neural-network training failure in which backpropagated gradients become extremely small across successive layers, limiting effective parameter updates.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionThe vanishing gradient problem is a training difficulty in which derivatives become extremely small and cause neural-network parameter updates to become ineffective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe difficulty in training deep networks when gradients become too small as they propagate backward through many layers.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA training failure mode in deep networks where gradient signals become too small to effectively update early layers.
• 

  liveBookmanning.com
  LLMs in ProductionA vanishing gradient is a training condition in which propagated derivatives become increasingly small and impede parameter updates.
• 

  liveBookmanning.com
  Data Without LabelsThe vanishing gradient problem is an optimization difficulty in which gradients become extremely small during learning, hindering the retention of information across many sequence steps.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A training difficulty in deep neural networks where gradients shrink as they propagate backward, hindering learning in early layers.

Compare how 14 books explain vanishing gradient problem →

Related concepts

• ReLUaddressed by · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• residual connectionaddressed by · A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
• LSTMaddressed by · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.
• ResNetaddressed by · ResNet is a family of deep neural-network architectures built from residual blocks with shortcut connections that facilitate information flow and optimization.
• GRUaddressed by · A gated recurrent unit (GRU) is a recurrent neural-network architecture that uses learned gates to regulate information flow and update a hidden state across sequence steps.
• normalizationaddressed by · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.