Residual connection

As explained in Graph Neural Networks in Action

A skip connection is a neural-network pathway that passes information around one or more intermediate layers to preserve or combine representations.

A residual connection, or skip connection, creates a pathway around intermediate GNN layers so information can bypass local processing and retain longer-range signals. section 5.5.2

Use residual connections when:

• Deep message passing causes over-smoothing, making node representations increasingly similar after roughly three or four layers. section 5.5.2
• Long-range interactions matter but stacking enough layers to match the problem radius would intensify over-squashing. section 5.5.2
• You need an alternative or complement to attention mechanisms, neighborhood sampling, added edges, or globally connected nodes. section 5.5.2

Figure 5.19 Visualization of over-squashing (Source: Alon and Yahav [7])

Visualize how information becomes compressed across graph distance and why bypass pathways can help preserve distant signals. section 5.5.2

How Graph Neural Networks in Action covers it

How deep it goes

Explains the motivation and trade-off, but stops short of specifying a layer-level implementation.

Examples

Illustrates the need with large molecular graphs, distant atoms, and toxicity-related properties.

What it assumes

Assumes familiarity with graph neural networks, message passing, and the problem radius.

What you can do after

Evaluate/understand when bypass pathways can reduce the need for excessive GNN depth.

How it connects

Connects residual connections to over-smoothing, over-squashing, problem radius, attention mechanisms, and GraphSAGE.

liveBookmanning.com

Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.
• 

  liveBookmanning.com
  Sutskever's ListA residual connection is a shortcut path that carries a block's input directly to an addition with the block's transformed output.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A direct link that transfers representations between corresponding layers or stages of a neural network.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA neural-network connection that bypasses intermediate layers and passes earlier activations to later layers.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA skip connection is a direct connection that carries an earlier layer's representation to a later corresponding layer, bypassing intervening layers.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA connection that adds a layer's input to its output so information and gradients can bypass one or more layers.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionA residual connection is a neural-network connection that combines a layer's input with its transformed output to facilitate information and gradient propagation.
• 

  liveBookmanning.com
  Transformers in ActionA residual connection adds a layer's input to its transformed output to facilitate optimization of deep neural networks.

Compare how 12 books explain residual connection →

Related concepts

• U-Netuses · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• ResNetused by · ResNet is a family of deep neural-network architectures built from residual blocks with shortcut connections that facilitate information flow and optimization.
• transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
• over-smoothingaddresses · Over-smoothing is a graph neural network failure mode in which repeated neighborhood aggregation or message passing makes node representations increasingly similar and difficult to distinguish.

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.