A skip connection gives a network an additional path that bypasses one or more layers. The ordinary feed-forward path remains, but an earlier activation also reaches a later layer, making deeper designs easier to optimize. section 6.3 section 6.5.1
Common skip-connection designs differ in how they combine paths:
- Concatenate activations: torch.cat([x, h], dim=1) changes feature shape from (B, D) and (B, H) to (B, D+H). section 6.3
- Send several earlier activations to one layer, increasing that layer's input size according to the number of incoming outputs. section 6.3
- Add paths in a residual block: h = x + F(x), where the short path leaves x unchanged and F(x) supplies learned complexity. section 6.5.1
The practical benefit is a shorter gradient path. Fewer operations reduce opportunities for gradient noise, helping information reach earlier layers and narrowing the gap between what a deep network can represent and what it can learn. Skip connections therefore support depth without forcing every signal through every operation. section 6.3 section 6.5.1
Implementing a fully connected skip block
A custom SkipFC module stores hidden layers and normalization layers in nn.ModuleList so PyTorch registers their parameters and includes them in automatic differentiation. A block can then be composed with nn.Sequential, for example: SkipFC(2, D, n), SkipFC(2, n, n), SkipFC(2, n, n), followed by nn.Linear(n, classes). section 6.3.1
Shape constraints in convolutional networks
A SkipConv2d module applies the same pattern with Conv2d and BatchNorm2d. Concatenation may change the channel axis, but tensors must match on width and height. Skipping across MaxPool2d fails because pooling changes spatial dimensions; use compatible shapes or adjust channels with a 1 × 1 convolution. section 6.3.2
Figure 6.5 Every operation done makes the network more complex, but also makes the gradient more complex, creating a tradeoff between depth (capacity) and learnability (optimization). Skip connections create a short path with fewer operations, and can make the gradient easier to learn. Only linear layers are shown for simplicity, but would also contain normalization and activation layers.
Compare ordinary, shortcut, and dense paths while tracing how fewer operations can simplify gradient learning. section 6.3
