After AlexNet’s eight layers, image-recognition networks grew deeper: VGGNet reached 19 layers and GoogLeNet 22. But adding layers eventually made some networks perform worse, not better. This is the degradation problem: greater depth offers more representational capacity, yet the resulting network can be harder to train. section 3.1 section 2.5.5
When deeper trains worse
On CIFAR-10, a plain 56-layer network had higher training error than a plain 20-layer network, as well as higher test error. The training-error gap is crucial: poor test performance was not simply a case of the deeper model fitting its training data too well. ImageNet comparisons likewise contrasted plain 18- and 34-layer networks with residual versions at the same depths. section 3.1
Figure 3.2 Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper 56-layer network shows a higher training error than the shallower 20-layer network, indicating that it is harder to optimize without additional architectural innovations. It also has a higher test error (right), showing that simply adding layers doesn’t improve performance. In both plots, the x-axis denotes training iterations in units of 10,000; each iteration corresponds to a single mini-batch update. As training progresses across approximately 60,000 iterations, the deeper network consistently underperforms. Used with permission granted by the lead author (Kaiming He).
The training and test curves show the 56-layer plain network persistently underperforming the 20-layer one as training proceeds. section 3.1
One way to picture the difficulty is a telephone game. As data passes through many transformations, small distortions can compound; during backpropagation, gradients can also shrink and lose useful information before reaching early layers. Vanishing gradients and degradation are closely related obstacles, though degradation names the worsening performance observed as depth increases. Batch normalization, careful initialization, and gradient clipping helped, but did not provide a direct route through dozens of nonlinear layers. section 3.1
ResNet’s response was a shortcut that carries an earlier signal around transformation layers and combines it with their output. Residual connections, followed by pre-activation, improved signal flow and made greater depth more useful. The lesson is practical: more compute or layers help only when the architecture can train them reliably. A shallow support vector machine has no comparable depth problem—or depth knob to turn. section 3.1 section 3.8
