Vanishing gradient problem

As explained in Graph Neural Networks in Action

An optimization problem in which gradients become progressively smaller as they are propagated through many computational steps, hindering learning of long-range dependencies.

The vanishing gradient problem occurs when gradients used by gradient descent approach zero as they pass through many recurrent steps or layers. Training then struggles to preserve the influence of earlier information, limiting an RNN’s effective memory for sequential prediction. section 6.2.2 section 6.5.1

Gated recurrent architectures address the problem through controlled memory updates:

• Use LSTM networks to introduce more complex dependencies than a vanilla RNN; their role is described, but their internals remain outside the treatment. section 6.2.2
• Use GRU update gates to control how much past information remains and reset gates to control how much is discarded. section 6.5.1
• Represent the GRU’s candidate activation as a proposed new state and its hidden state as the resulting representation. section 6.5.1

PyTorch GRU for pose estimation

A practical replacement for a recurrent baseline defines nn.GRU(input_size, hidden_size, num_layers, batch_first=True), initializes a zero hidden state on the input device, runs the sequence, and maps the final selected outputs through a linear layer. The same workflow supports predicting sensor trajectories over time. section 6.5.1 section 6.2.2

Figure 6.18 Design of the GRU layer, where r_t represents the reset gate, z_t is the update gate, ~h_t is the candidate function, and h_t is the final actual hidden state

Inspect the GRU diagram to connect reset gates, update gates, candidate activation, and the final hidden state. section 6.5.1

How Graph Neural Networks in Action covers it

How deep it goes

Defines gradient decay and explains GRU gate roles, but defers the underlying causes and LSTM internals.

Examples

Uses sensor-trajectory prediction, pose-estimation RNN and GRU definitions, and GRU gate diagrams as concrete teaching artifacts.

What you can do after

Build it from scratch using PyTorch GRU layers with initialized hidden states and linear output projections for sequence prediction.

How it connects

Connects gradient descent with recurrent neural networks, LSTM networks, GRU networks, hidden states, and sequential data.

liveBookmanning.com

Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListA vanishing gradient is an optimization failure in which derivatives become too small to support effective parameter updates during neural-network training.
• 

  liveBookmanning.com
  Grokking Machine LearningA neural-network training problem in which gradients become extremely small as they are propagated through layers, slowing or preventing learning.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe training difficulty in deep networks where repeated multiplication of small derivatives makes early-layer gradients nearly zero.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA vanishing gradient is a training condition in which gradients become extremely small as they propagate through a deep network, slowing or preventing parameter updates in earlier layers.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training problem in deep networks where gradient signals become too small to effectively update earlier layers.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA training instability in which gradients become too small for effective learning updates.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA neural-network training failure in which backpropagated gradients become extremely small across successive layers, limiting effective parameter updates.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionThe vanishing gradient problem is a training difficulty in which derivatives become extremely small and cause neural-network parameter updates to become ineffective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe difficulty in training deep networks when gradients become too small as they propagate backward through many layers.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA training failure mode in deep networks where gradient signals become too small to effectively update early layers.
• 

  liveBookmanning.com
  LLMs in ProductionA vanishing gradient is a training condition in which propagated derivatives become increasingly small and impede parameter updates.
• 

  liveBookmanning.com
  Data Without LabelsThe vanishing gradient problem is an optimization difficulty in which gradients become extremely small during learning, hindering the retention of information across many sequence steps.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A training difficulty in deep neural networks where gradients shrink as they propagate backward, hindering learning in early layers.

Compare how 14 books explain vanishing gradient problem →

Related concepts

• ReLUaddressed by · ReLU, or rectified linear unit, is an activation function that maps an input x to max(0, x).
• residual connectionaddressed by · A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
• LSTMaddressed by · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.
• ResNetaddressed by · ResNet is a family of deep neural-network architectures built from residual blocks with shortcut connections that facilitate information flow and optimization.
• GRUaddressed by · A gated recurrent unit (GRU) is a recurrent neural-network architecture that uses learned gates to regulate information flow and update a hidden state across sequence steps.
• normalizationaddressed by · Normalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.