chapter three

3 ResNet Revolution

 

This chapter covers

  • The challenges of training deep neural networks
  • Residual connections and how they revolutionized deep learning
  • Evolution to ResNet v2 and training networks exceeding 1,000 layers
  • Pooling and striding were replaced with dilated convolutions for dense prediction tasks
  • CS231n becomes the first deep learning course at Stanford
Papers
  • Deep Residual Learning for Image Recognition (2015) He, Zhang, Ren, and Sun
  • Identity Mappings in Deep Residual Networks (2016) He, Zhang, Ren, and Sun
  • Multi-Scale Context Aggregation by Dilated Convolutions (2016) Yu and Koltun
  • Stanford CS231n (2015)

Before 2015, increasing depth did not immediately translate into improved performance. In fact, deeper models often exhibited higher training error. The heart of the problem was a double bind. Depth was required for representational capacity, yet depth impaired optimization; reducing depth alleviated optimization problems, but capped capacity. Just as despair seemed justified, a deceptively simple idea emerged.

Residual connections ameliorate this tension by providing shortcuts that preserve the forward signal and maintain effective gradient pathways during backpropagation. Operationalized in the Residual Network (ResNet), it earns its place on Sutskever’s List because it represents a pivotal shift in design philosophy in which “add a residual connection” has become an indispensable tool in the machine learning toolkit.[1]

3.1 Telephone Game

3.2 Comparison with Other Major Architectures

3.2.1 ResNet vs. AlexNet & ZFNet

3.2.2 ResNet vs. VGGNet

3.2.3 ResNet vs. GoogLeNet

3.3 Real-World Adoption

3.4 ResNet-v2

3.4.1 Post-Activation to Pre-Activation

3.4.2 Ablation Studies

3.5 Extending ResNet with Dense Prediction

3.6 Human Measuring Sticks

3.7 CS231n

3.8 Scale What?