The vanishing gradient problem occurs when gradients become extremely small as they move backward through a neural network. Backpropagation then makes tiny weight updates, slowing learning and potentially delaying a useful classifier. section 10.12 section 10.25
Why sigmoid layers can stall learning
The sigmoid curve is very flat for large positive and negative inputs, so its derivatives are close to zero. Backpropagation composes many such derivatives across layers; the resulting steps become tiny, and reaching a good classifier may take a very long time. section 10.12
Common activation-function responses include:
- Use tanh as an improvement over sigmoid, while recognizing that tanh can still suffer from vanishing gradients. section 10.14
- Use ReLU for hidden layers: it returns zero for negative inputs, preserves nonnegative inputs, and has derivative 1 for positive inputs. section 10.14 section 10.25
- Combine activations, commonly using ReLU internally and sigmoid at the final classification layer. section 10.14
Figure 10.17 The sigmoid function is flat at the ends, which means that for large positive and negative values, its derivative is very small, hampering the training.
Use the sigmoid graph to see how flat ends produce very small derivatives and hamper training. section 10.12
















