Degradation

As explained in Math and Architectures of Deep Learning

In residual networks, the phenomenon where adding more layers makes training and test performance worse instead of better.

In residual networks, degradation occurs when increasing depth makes accuracy saturate and then decline. A 56-layer network can have higher training and test error than a 20-layer network on CIFAR-10, despite having greater expressive power. Higher training error distinguishes degradation from ordinary overfitting. section 11.2.3

A deeper network should at least match a shallower one by preserving the original layers and learning the identity function in the added layers. In practice, the added layers struggle to learn identity mappings. For a target function h(x), learning the residual g(x) = h(x) − x can be simpler than learning h(x) directly. section 11.2.3

The degradation diagnosis motivates residual architecture:

• Rejects overfitting and unhealthy gradients as sufficient explanations for the observed result. section 11.2.3
• Uses shortcut or skip connections so layers can more easily preserve an identity mapping. section 11.2.3

Figure 11.9 Training error (left) and test error (right) on the CIFAR-10 data set with 20-layer and 56-layer networks. (Source: “Deep residual learning for image recognition”; https://arxiv.org/pdf/1512.03385.pdf.)

Compare training and test error curves for 20-layer and 56-layer networks on CIFAR-10. section 11.2.3

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines the phenomenon, compares errors, diagnoses likely causes, and derives skip connections without detailing optimization internals.

Examples

Compares 20-layer and 56-layer CIFAR-10 networks and motivates residual blocks implemented with PyTorch.

What you can do after

Evaluate depth-related error increases and understand why residual skip connections address them.

How it connects

Connects network depth, overfitting, vanishing gradients, identity functions, residual learning, and skip connections.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListModel degradation is the worsening of training or evaluation performance as network depth increases despite added representational capacity.
• 

  liveBookmanning.com
  LLMs in ProductionPerformance decay is a decline in a model's output quality over time despite the model continuing to operate.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA decline in a deployed machine learning model's predictive or operational performance over time.
• 

  liveBookmanning.com
  Building Reliable AI SystemsA decline in the quality of a system's outputs compared with an earlier or expected level.

Compare how 5 books explain degradation →

Related concepts

• data driftproduced by · Data drift is a change over time in the statistical distribution or characteristics of data relative to a reference distribution or set of characteristics.
• throughputuses · The amount of work, data, or output a system processes or completes per unit of time.
• atomicityproduced by · Atomicity is the property that an operation or group of operations takes effect as an indivisible, all-or-nothing unit, without observable partial effects.
• benchmarkmeasured by · A benchmark is a defined task, dataset, workload, procedure, or reference measure used to assess or compare performance or capability under specified criteria or conditions.
• problemkind of · A problem is a difficulty, obstacle, or undesirable condition requiring investigation, mitigation, or resolution.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.