liveBook · Manning

Vanishing gradient problem

As explained in Grokking Machine Learning, Second Edition

A neural-network training problem in which gradients become extremely small as they propagate through layers, substantially slowing learning.

The vanishing gradient problem occurs when gradients become extremely small as they move backward through a neural network. Backpropagation then makes tiny weight updates, slowing learning and potentially delaying a useful classifier. section 10.12 section 10.25

Why sigmoid layers can stall learning

The sigmoid curve is very flat for large positive and negative inputs, so its derivatives are close to zero. Backpropagation composes many such derivatives across layers; the resulting steps become tiny, and reaching a good classifier may take a very long time. section 10.12

Common activation-function responses include:

  • Use tanh as an improvement over sigmoid, while recognizing that tanh can still suffer from vanishing gradients. section 10.14
  • Use ReLU for hidden layers: it returns zero for negative inputs, preserves nonnegative inputs, and has derivative 1 for positive inputs. section 10.14 section 10.25
  • Combine activations, commonly using ReLU internally and sigmoid at the final classification layer. section 10.14
Figure 10.17 The sigmoid function is flat at the ends, which means that for large positive and negative values, its derivative is very small, hampering the training.

Use the sigmoid graph to see how flat ends produce very small derivatives and hamper training. section 10.12

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains sigmoid derivatives, repeated composition during backpropagation, and activation-function remedies without developing deeper optimization theory.
Examples
Uses sigmoid’s flat graph, repeated sigmoid compositions, tanh, ReLU, and a mixed-activation classification architecture.
What it assumes
Uses neural-network layers, sigmoid derivatives, and backpropagation as working context.
What you can do after
Apply ReLU in hidden layers and reserve sigmoid for classification outputs to reduce vanishing-gradient effects.
How it connects
Connects sigmoid, hyperbolic tangent, ReLU, activation functions, backpropagation, and neural-network training.

Also explained in

Compare how 17 books explain vanishing gradient problem →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.