chapter three

3 ResNet Revolution

 

This chapter covers

  • The challenges of training deep neural networks
  • Residual connections and how they revolutionized deep learning
  • Evolution to ResNet v2 and training networks exceeding 1,000 layers
  • Pooling and striding were replaced with dilated convolutions for dense prediction tasks.
  • CS231n becomes the first deep learning course at Stanford
Papers
  • Deep Residual Learning for Image Recognition (2015) He, Zhang, Ren, and Sun
  • Identity Mappings in Deep Residual Networks (2016) He, Zhang, Ren, and Sun
  • Multi-Scale Context Aggregation by Dilated Convolutions (2016) Yu and Koltun
  • Stanford CS231n (2015) Fei-Fei Li and Andrej Karpathy

Before 2015, increasing depth did not immediately translate into improved performance. In fact, deeper models often exhibited higher training error. The heart of the problem was a double bind. Depth was required for representational capacity, yet it impaired optimization. Reducing depth eased optimization but capped capacity. Just as despair seemed justified, a deceptively simple idea emerged.

Residual connections ease this tension by providing shortcuts that preserve the forward signal and maintain effective gradient pathways during backpropagation. Operationalized in the Residual Networks (ResNets), they earn their place on Sutskever’s List because they represent a pivotal shift in design philosophy in which “add a residual connection” has become an indispensable tool in the machine learning toolkit.[1]

3.1 Telephone Game

3.2 Comparison with Other Major Architectures

3.2.1 ResNet vs. AlexNet & ZFNet

3.2.2 ResNet vs. VGGNet

3.2.3 ResNet vs. GoogLeNet

3.3 Real-World Adoption

3.4 ResNet-v2

3.4.1 Post-Activation to Pre-Activation

3.4.2 Ablation Studies

3.5 Extending ResNet with Dense Prediction

3.6 Human Measuring Sticks

3.7 CS231n

3.8 Scale What?