In a very deep network, training can fail when gradients cannot travel effectively through the layers. Residual connections offer shorter routes: gradients can pass along skip connections instead of traversing every layer in sequence. This helps explain why a 1,000-layer ResNet can behave less like one long chain and more like a network with many shorter paths. section 3.4.2
ResNet v2 made the skip route a clean identity mapping by moving normalization and activation before convolutions and removing the activation after the paths merge. The change avoids placing an activation-function bottleneck on the shared route, improving gradient flow in very deep networks. section 3.4
Short paths in a 110-layer ResNet
Layer-removal experiments found that removing individual residual layers caused only modest error increases. Most effective gradient flow followed subnetworks of roughly 10–34 layers; very deep paths contributed little and often learned identity mappings. section 3.4.2
Path length matters in sequence models too. Reversing input sentences brought source words relevant to early predictions closer to the decoder’s initial state, making their relationship easier to learn through backpropagation. Later methods, including attention, reduced the need for this reversal trick. section 5.7
Figure 5.10 The reversal trick shortens the path from relevant source tokens to the decoder’s first predictions. In a vanilla RNN encoder-decoder without attention (a), the decoder starts from the encoder’s final state. Reversing the source (b) makes tokens that matter for early outputs appear later in the encoder’s processing, so their influence reaches the decoder more directly.
Compare how reversing the input shortens the route from relevant source words to the decoder’s first predictions. section 5.7
