Residual connection

As explained in Grokking AI Algorithms, Second Edition

A skip connection is a direct connection that carries an earlier layer's representation to a later corresponding layer, bypassing intervening layers.

A residual connection is a shortcut that carries an earlier representation past some processing and combines it with the later result. Think of keeping the original sketch while adding a new layer of detail: the later layer need not recreate everything from memory. section 11.26

In a Transformer feed-forward block, the shortcut carries the block’s input directly to its output. The processed result is added to that input, so both must have the same size. A project-down layer turns an 8D intermediate vector back into a 4D vector before the addition. This helps stabilize training and lets the model bypass feed-forward calculations that are not useful for a particular token. section 11.26

Following the Transformer shortcut

After projection, one 4D result is [1.6265, -1.3635, -0.2657, 1.9661]. Adding the bias [0.05, -0.1, 0.15, -0.02] gives [1.6765, -1.4635, -0.1157, 1.9461]. The residual step then adds the feed-forward block’s original 4D input to this 4D result, position by position. section 11.26

A U-Net uses a related shortcut for a different job. Downsampling captures what an image contains but loses precise information about where details belong. A skip connection carries an early, high-resolution feature map directly to the corresponding decoder layer. Here the maps are concatenated, not added: an 8×8×8 early tensor joined with a 16×8×8 upsampled tensor becomes a 24×8×8 tensor. The decoder can then use both spatial detail and deeper, abstract information when reconstructing an image. section 12.15 section 12.19

12.42 How skip-connection feature maps are combined with the upsampled feature maps

Follow the early feature maps across to the decoder and see where they join the upsampled maps. section 12.19

How Grokking AI Algorithms, Second Edition covers it

How deep it goes

Works through matching vector sizes and residual addition, then traces U-Net concatenation with tensor shapes.

Examples

Traces an 8D-to-4D projection and an 8×8×8 feature map concatenated with a 16×8×8 upsampled map.

What you can do after

Build it from scratch by adding matching Transformer vectors or concatenating U-Net feature maps before convolution.

How it connects

Connects residual shortcuts to Transformer feed-forward blocks and skip connections to U-Net encoders, decoders, and upsampling.

liveBookmanning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.
• 

  liveBookmanning.com
  Sutskever's ListA residual connection is a shortcut path that carries a block's input directly to an addition with the block's transformed output.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A direct link that transfers representations between corresponding layers or stages of a neural network.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA neural-network connection that bypasses intermediate layers and passes earlier activations to later layers.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA connection that adds a layer's input to its output so information and gradients can bypass one or more layers.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionA skip connection is a neural-network pathway that passes information around one or more intermediate layers to preserve or combine representations.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionA residual connection is a neural-network connection that combines a layer's input with its transformed output to facilitate information and gradient propagation.
• 

  liveBookmanning.com
  Transformers in ActionA residual connection adds a layer's input to its transformed output to facilitate optimization of deep neural networks.

Compare how 12 books explain residual connection →

Related concepts

• U-Netuses · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
• ResNetused by · ResNet is a family of deep neural-network architectures built from residual blocks with shortcut connections that facilitate information flow and optimization.
• transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
• over-smoothingaddresses · Over-smoothing is a graph neural network failure mode in which repeated neighborhood aggregation or message passing makes node representations increasingly similar and difficult to distinguish.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.