Before 2015, adding layers to a convolutional network could make it worse even on the training data. A 56-layer plain network, for example, had higher training error than a 20-layer one on CIFAR-10. This was not simply a case of a large model memorizing its training set and failing elsewhere: the deeper model was harder to optimize. Microsoft researchers answered this depth problem with ResNet, introduced in “Deep Residual Learning for Image Recognition.” section 3.0 section 3.1
Figure 3.2 Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper 56-layer network shows a higher training error than the shallower 20-layer network, indicating that it is harder to optimize without additional architectural innovations. It also has a higher test error (right), showing that simply adding layers doesn’t improve performance. In both plots, the x-axis denotes training iterations in units of 10,000; each iteration corresponds to a single mini-batch update. As training progresses across approximately 60,000 iterations, the deeper network consistently underperforms. Used with permission granted by the lead author (Kaiming He).
Compare the training-error curves first: the 56-layer plain network underperforms the 20-layer network despite having more layers. The test-error curves show that added depth also fails to improve the result. section 3.1
A residual block gives information two routes. One transforms the input through layers that learn features; the other carries the input around those layers. The routes are added together. Think of a telephone game with occasional written notes passed ahead: even if the whispered message changes, part of the original arrives intact. In a network, that shortcut helps preserve forward signals and gives gradients a clearer route backward during training. It addresses the practical tension between needing depth for representational capacity and struggling to train that depth. section 3.1 section 3.0
Figure 3.1 Diagram of a simple ResNet block. It illustrates how input data flows along two paths: one through two transformation layers with activation steps, and another around via a direct “shortcut” path with no transformations. The outputs of these two paths are combined, and the resulting vector undergoes a final activation step. This “shortcut” helps the network learn more effectively by preserving information from earlier layers. Used with permission granted by the lead author (Kaiming He).
Follow the input through the transformation layers and along the direct shortcut, then locate where their outputs are combined. In this original block, an activation follows the addition—a detail later changed in ResNet v2. section 3.1 section 3.4
ResNet made depth easier to scale without redesigning a new arrangement of branches for every model. VGGNet had also pursued depth by repeating small convolutional layers, but very deep stacks remained difficult to train. GoogLeNet used parallel paths to capture features at different scales, at the cost of manually balancing their design. ResNet instead repeated residual blocks across variants including ResNet-18, -34, -50, -101 and -152. Its basic block uses two 3 × 3 convolutions; deeper variants use a bottleneck that compresses and restores feature dimensions around a 3 × 3 layer. section 3.2.2 section 3.2.3 section 3.1
The gains were measurable. At the 2015 ImageNet competition, a single ResNet reached 4.49% top-5 error; an ensemble reached 3.57%. Residual networks also became useful components of larger vision systems. Replacing VGG-16 with ResNet-101 in Faster R-CNN improved accuracy by 6% to 7% on COCO, and later systems used ResNet variants for tasks including medical-image analysis and driving perception. section 3.0 section 3.3
The shortcut still had a weakness: original ResNet applied ReLU after adding the two paths, so the combined signal did not pass onward unchanged. “Identity Mappings in Deep Residual Networks” introduced ResNet v2, moving batch normalization and activation before the convolutions and removing the post-addition activation. Ablation studies favored this full pre-activation design over several partial changes. On CIFAR-10, a 1,001-layer v2 network reached 4.62% classification error, compared with 7.93% for an original-design 1,202-layer network. The lesson is not that depth alone guarantees accuracy; an architecture must let useful signals survive it. section 3.4 section 3.4.2 section 3.8
