Why can adding layers make a neural network harder to train? In comparisons of 18- and 34-layer plain networks, the deeper network performed worse even during training. The puzzle is that extra layers should, in principle, be able to leave an already useful computation alone. An identity function does exactly that: it returns its input unchanged. section 3.1
A residual block gives the input two routes. One passes through layers that learn a transformation; the other is a shortcut that carries the input unchanged. Adding the routes lets the block learn an adjustment to its input rather than a wholly new output. If no adjustment helps, the transformed route can contribute nothing. That is the appeal of identity mapping when building deeper networks. section 3.1
Figure 3.1 Diagram of a simple ResNet block. It illustrates how input data flows along two paths: one through two transformation layers with activation steps, and another around via a direct “shortcut” path with no transformations. The outputs of these two paths are combined, and the resulting vector undergoes a final activation step. This “shortcut” helps the network learn more effectively by preserving information from earlier layers. Used with permission granted by the lead author (Kaiming He).
The block diagram separates the transformation route from the direct shortcut, then shows where their outputs meet. section 3.1
An unchanged shortcut does not necessarily make the entire block an exact identity function. In the original ResNet design, a ReLU activation followed the addition and could alter the combined signal. ResNet v2 moved normalization and activation before the convolutions and removed that post-addition activation, keeping a cleaner route for information and gradients through very deep networks. section 3.4
Why leave the shortcut alone?
Ablation studies tested seemingly modest alternatives: scaling the shortcut by 0.5, dropping it randomly, adding a learnable gate, or inserting convolutions. These changes degraded performance or convergence. Placing batch normalization after the addition also hurt accuracy. The practical lesson is that a shortcut meant to preserve input is most useful when it remains unaltered. section 3.4.2
