Before ResNet, adding layers could make a neural network harder to train rather than more accurate. A deeper model had greater capacity, yet its training error could rise: forward signals became distorted and gradients weakened on their way back. In 2015, Microsoft researchers addressed this problem with a shortcut that lets information bypass a group of layers. section 3.0 section 3.1
A residual connection carries a block’s input along two paths. One path transforms it through layers such as convolutions, batch normalization, and activations; the other carries it forward unchanged. The paths are added together. Like passing a written note past several players in a telephone game, the shortcut preserves a clear version of the original message while the other path makes useful changes. It also gives gradients a direct route backward. section 3.1
Figure 3.1 Diagram of a simple ResNet block. It illustrates how input data flows along two paths: one through two transformation layers with activation steps, and another around via a direct “shortcut” path with no transformations. The outputs of these two paths are combined, and the resulting vector undergoes a final activation step. This “shortcut” helps the network learn more effectively by preserving information from earlier layers. Used with permission granted by the lead author (Kaiming He).
The block diagram shows the transformed path, the direct shortcut, and their addition. section 3.1
Depth without the shortcut
On CIFAR-10, a plain 56-layer network had higher training and test error than a plain 20-layer network. On ImageNet, comparisons of 18- and 34-layer plain networks with equally deep ResNets showed why adding layers was not enough: the architecture also had to make those layers trainable. section 3.1
The shortcut made a repeatable residual block practical to stack. Unlike VGGNet’s repeated convolutions, which became difficult to train beyond 19 layers, ResNet scaled across substantially deeper designs without redesigning the basic idea. Its 2015 ImageNet results helped make residual networks a common backbone for later models. section 3.2.2 section 3.2.3
The shortcut’s exact form matters. Original ResNet applied an activation after adding the two paths; in very deep networks, that could interfere with signal flow. ResNet v2 moved normalization and activation before the convolutions and removed the post-addition activation, leaving a clean identity path. Ablation studies found that scaling, gating, randomly dropping, or adding convolutions to that path generally hurt convergence or accuracy. The practical lesson is not simply to add another route, but to keep it unobstructed. section 3.4 section 3.4.2
