The unstable gradient problem occurs when gradients lose a useful scale during backpropagation. Products above 1 produce exploding gradients; products below 1 produce vanishing gradients, so early layers learn at dramatically different speeds or not at all. section 8.2.2
Common responses include:
- Use Xavier or He initialization to adjust random weights according to the previous layer’s size. section 8.2.2
- Use ReLU to help keep gradient updates stable during training. section 8.2.2
