How deeply does Math and Architectures of Deep Learning cover ResNet?

Connects degradation to identity-function learning, then descends into residual blocks, downsampling, and PyTorch ResNet-34 construction.

ResNet

As explained in Math and Architectures of Deep Learning

A deep convolutional neural network architecture that uses identity shortcut connections to ease optimization of very deep networks.

ResNet is a convolutional neural network architecture designed to make very deep networks trainable. Simply stacking layers can produce the degradation problem: deeper networks may have higher training and test error than shallower ones, even without vanishing or exploding gradients. section 11.2.3

A deeper network should be able to reproduce a shallower network by learning the identity function in its added layers, but ordinary layers struggle to learn identity mappings. ResNet adds shortcut connections so layers can learn a residual transformation instead: if h(x) is the desired mapping, the stack learns g(x) = h(x) − x while the shortcut carries x directly. section 11.2.3 section 11.5

Why the shortcut matters

On CIFAR-10, a 56-layer network can have higher training and test error than a 20-layer network. The residual design addresses this counterintuitive degradation by making the identity behavior available through a shortcut rather than forcing every added layer to learn it. section 11.2.3

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Sutskever's ListResNet is a deep neural network architecture that uses residual connections to facilitate the training of very deep networks.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA neural network architecture built from residual blocks with skip connections that help information and gradients propagate through deep networks.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA family of residual neural network architectures that use skip connections to ease training of deep models.

Compare how 4 books explain ResNet →

Related concepts

residual connectionuses · A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
vision modelkind of · A vision model is a machine-learning model that processes visual data, such as images or video, to analyze, generate, or extract information.
Faster R-CNNused by · Faster R-CNN is an object-detection architecture that uses a learned region proposal network and detection heads to localize and classify objects in images.
ResNet50includes · ResNet50 is a 50-layer residual convolutional neural network architecture for visual recognition and image processing.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.