Training from scratch begins with newly initialized weights. Random values let neurons learn different features; identical values, such as zero, give neurons identical gradients and updates, collapsing a layer into a single unit. section 2.4
Initialization can also destabilize learning: weights that are too large can cause exploding gradients or saturation, while weights that are too small can make gradients vanish. In deep networks, vanishing gradients leave earlier layers with updates too small to learn effectively. These difficulties once led researchers to suspect depth itself, even though GPUs could accelerate training without making optimization easier. section 2.4
Figure 2.2 The challenge of optimizing functions shaped like long, narrow valleys. The contour lines represent a valley, with the optimal direction of progress indicated by an arrow running along the valley’s base. Smaller arrows represent steps taken by gradient descent: the left diagram shows large steps oscillating inefficiently across the valley, while the right diagram shows small steps making slow progress along the base. The “pathological” nature does not arise from high or low curvature, but from the combination of high curvature across the valley and low curvature along it. Used with permission granted by the author (James Martens).
The narrow-valley illustration shows how gradient descent can oscillate across a valley or make slow progress along its base. section 2.4
