An ablation study removes or modifies model components to measure their individual contributions. In ResNet, experiments on skip connections tested whether small architectural changes helped or hindered training. section 3.4.2
ResNet experiments compared:
- Identity skip connections with scaled, randomly dropped, gated, or convolutional alternatives; these alternatives degraded performance or disrupted optimization. section 3.4.2
- Different placements of ReLU and batch normalization; the full pre-activation strategy performed best, placing both before convolutions and omitting activations after addition. section 3.4.2
