liveBook · Manning

Residual connection

Also called skip connection, shortcut connections

A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations so it can be combined with a later computation.

How 18 Manning books explain it

Inside Deep Learning: Math, Algorithms, Models

A bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.

How it is used
Provides a short route around learned layers to make deeper fully connected and convolutional networks easier to train.
How deep it goes
Derives additive paths and shape constraints, then implements same-channel blocks and channel-changing bottlenecks as PyTorch modules.
Examples
Uses PyTorch residual blocks and bottlenecks, a Fashion-MNIST CNN comparison, and an exercise varying residual depth.
What you can do after
Build it from scratch as PyTorch residual blocks and bottlenecks, choosing an identity or channel-adjusting shortcut.

Read how Inside Deep Learning: Math, Algorithms, Models explains residual connection →

Sutskever's List

A residual connection is a shortcut path that carries a block's input directly to an addition with the block's transformed output.

How it is used
Addresses the optimization difficulty that arose when greater network depth increased training error.
How deep it goes
Explains forward and backward paths, compares plain networks with ResNets, and examines pre-activation design through ablations.
Examples
Uses a telephone-game shortcut, CIFAR-10 depth comparisons, and tests that alter the identity path.
What you can do after
Build it from scratch as a two-path block, or diagnose whether activation and skip-path changes obstruct signal flow.

Read how Sutskever's List explains residual connection →

Build a Text-to-Image Generator (from Scratch)

A direct link that transfers representations between corresponding layers or stages of a neural network.

How it is used
Preserves information across transformed layers in transformers and ResNet50, and across matching encoder–decoder stages in denoising U-Nets.
How deep it goes
Defines additive residual connections and shows two implemented transformer additions; describes U-Net feature merging without specifying its merge operation.
Examples
Shows a transformer encoder block with two additions, ResNet50 bottleneck blocks, and a denoising U-Net with matching encoder–decoder links.
What you can do after
Build it from scratch in a transformer encoder block by adding each attention or feed-forward output to its sublayer input.

Read how Build a Text-to-Image Generator (from Scratch) explains residual connection →

Learn Generative AI with PyTorch

A neural-network connection that bypasses intermediate layers and passes earlier activations to later layers.

How it is used
Uses residual connections in image generators and Transformer sublayers to support information flow and stable training.
How deep it goes
Explains the input-plus-transformation operation and shows its implementation in convolutional and decoder blocks.
Examples
Uses a nine-block CycleGAN generator, GPT-2XL decoder, six-block music Transformer, and U-Net concatenation contrast.
What you can do after
Build residual additions into convolutional blocks and Transformer decoder sublayers using PyTorch.

Read how Learn Generative AI with PyTorch explains residual connection →

Build AI Drug Discovery Pipelines

A residual connection adds a layer's input representation to that layer's transformed output.

How it is used
Uses residual connections in an encoder-only transformer to preserve residue information as attention incorporates sequence context.
How deep it goes
Explains input-output addition, gradient flow, and subsequent layer normalization without a code implementation.
Examples
Illustrates preservation of an alanine’s identity alongside helix context in a protein-residue representation.
What you can do after
Evaluate and understand how residual addition supports information retention and gradient flow in a transformer encoder.

Read how Build AI Drug Discovery Pipelines explains residual connection →

Deep Learning with Python, Third Edition

How it is used
Uses shortcuts to support deeper ConvNets and preserve image detail in a denoising U-Net.
How deep it goes
Explains gradient flow and shape matching, then implements additive blocks and identifies concatenative shortcuts.
Examples
Demonstrates projected pooling blocks, a dogs-versus-cats classifier, and a denoising autoencoder.
What you can do after
Apply it via Keras by adding residual paths and projecting them when block outputs change shape.

Read how Deep Learning with Python, Third Edition explains residual connection →

CUDA for LLMs

A residual connection adds a sublayer's input to its output, providing a direct path through the network.

How it is used
Places residual connections around attention and feed-forward computations in stacked transformer blocks.
How deep it goes
Shows both additions in a block forward pass but does not detail a residual-specific CUDA kernel.
Examples
Uses a PyTorch block with separate attention and feed-forward residual additions.
What you can do after
Evaluate and understand how bypass paths combine with normalized sublayer outputs in a transformer block.

Read how CUDA for LLMs explains residual connection →

Deep Learning with PyTorch, Second Edition

A connection that bypasses one or more network layers to provide an alternate path for activations or gradients.

How it is used
Uses shortcut paths to make deep convolutional networks and transformer decoders easier to train.
How deep it goes
Explains vanishing gradients and implements a convolutional residual block, including normalization and initialization.
Examples
Illustrates shortcuts with a three-convolution network, a reusable residual block, and a GPT-style decoder diagram.
What you can do after
Builds a simple residual block in PyTorch and identifies summed shortcuts in a transformer decoder.

Read how Deep Learning with PyTorch, Second Edition explains residual connection →

Grokking AI Algorithms, Second Edition

A skip connection is a direct connection that carries an earlier layer's representation to a later corresponding layer, bypassing intervening layers.

How it is used
Uses shortcuts to stabilize Transformer training and preserve spatial detail during U-Net image reconstruction.
How deep it goes
Explains shortcut purposes and traces addition and concatenation with vector and tensor dimensions.
Examples
Traces a four-dimensional Transformer residual and combines 16 and eight U-Net feature maps into a 24-channel tensor.
What you can do after
Evaluate and understand how shortcut combinations retain earlier information in Transformer and U-Net architectures.

Read how Grokking AI Algorithms, Second Edition explains residual connection →

Math and Architectures of Deep Learning

A connection that adds a layer's input to its output so information and gradients can bypass one or more layers.

How it is used
Explains how ResNet trains deeper convolutional networks when simply stacking layers causes degradation.
How deep it goes
Defines residual learning, explains identity shortcuts, and reaches PyTorch block and architecture implementations.
Examples
Uses CIFAR-10 depth comparisons, residual-block diagrams, PyTorch BasicBlock code, and ResNet-34 construction.
What you can do after
Builds residual blocks in PyTorch and assembles or invokes a ResNet-34 model.

Read how Math and Architectures of Deep Learning explains residual connection →

Deep Learning with R, Third Edition

A residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.

How it is used
Improves deep convolutional-network trainability by preserving representations and allowing gradients to bypass noisy processing blocks.
How deep it goes
Explains the shortcut mechanism, shape-matching projections, pooling case, and a reusable stacked-block implementation.
Examples
Uses R/Keras pseudocode, channel-changing blocks, pooling blocks, and a three-block convolutional model.
What you can do after
Build residual convolutional blocks with shape-compatible shortcuts and stack them into deeper image-classification models.

Read how Deep Learning with R, Third Edition explains residual connection →

Applied Reinforcement Learning

A neural-network connection that adds a layer's input directly to its transformed output.

How it is used
Preserves an input representation while an attention or feedforward sub-layer adds new information.
How deep it goes
Defines the addition as x + SubLayer(x), explains preserved information and direct gradient flow, and stops short of implementation details.
Examples
Uses Transformer encoder layers that wrap self-attention and feedforward sub-layers processing zone representations.
What you can do after
Evaluate/understand how adding a layer input to its output can preserve information and ease deep-network training.

Read how Applied Reinforcement Learning explains residual connection →

Building Specialized AI Systems

A residual connection adds a module's input to its output to support information and gradient flow through a neural network.

How it is used
Supports information and gradient flow while Transformer blocks process representations during training and fine-tuning.
How deep it goes
Defines the shortcut and locates it within Transformer blocks, but does not detail its numerical computation.
Examples
Shows residual connections in encoder and decoder stacks through the Add & LN pattern.
What you can do after
Understand where residual connections fit within a Transformer block and why shortcuts support model optimization.

Read how Building Specialized AI Systems explains residual connection →

Rearchitecting LLMs

A residual connection adds a module's input representation to its transformed output so information can bypass the transformation.

How it is used
Uses residual-inclusive input/output similarity to identify attention modules that may be removed.
How deep it goes
Explains hook locations and why excluding the residual can distort attention importance scores.
Examples
Uses Llama and Gemma decoder-layer hook points and contrasts importance measurement with versus without the residual.
What you can do after
Evaluate attention-module importance by comparing input and output cosine similarity with the residual included.

Read how Rearchitecting LLMs explains residual connection →

Graph Neural Networks in Action

A skip connection is a neural-network pathway that passes information around one or more intermediate layers to preserve or combine representations.

How it is used
Uses skip connections to preserve long-range information in graph neural networks without relying solely on deeper message passing.
How deep it goes
Explains the motivation and trade-off, but stops short of specifying a layer-level implementation.
Examples
Illustrates the need with large molecular graphs, distant atoms, and toxicity-related properties.
What you can do after
Evaluate/understand when bypass pathways can reduce the need for excessive GNN depth.

Read how Graph Neural Networks in Action explains residual connection →

Knowledge Graphs and LLMs in Action

A residual connection is a neural-network connection that combines a layer's input with its transformed output to facilitate information and gradient propagation.

How it is used
Improves information and gradient flow across graph neural-network message-passing layers.
How deep it goes
Explains shortcut motivation and a concatenation-based update, while deferring detailed residual variants and broader architecture design.
Examples
Uses a GraphSAGE update function that concatenates node and aggregated-neighbor features before applying a linear transformation and ReLU.
What you can do after
Apply residual-style updates in GNNs to preserve features and support deeper message-passing models.

Read how Knowledge Graphs and LLMs in Action explains residual connection →

Transformers in Action

A residual connection adds a layer's input to its transformed output to facilitate optimization of deep neural networks.

How it is used
Supports optimization and information preservation within stacked transformer encoder sublayers.
How deep it goes
Explains the addition, normalization order, dropout placement, and encoder-layer use, but stops short of optimization mathematics.
Examples
Demonstrates residual additions around attention and feed-forward outputs in a simplified encoder-layer listing.
What you can do after
Build it from scratch by adding dropout-regularized sublayer outputs to their inputs before applying layer normalization.

Read how Transformers in Action explains residual connection →

Quantization and Fast Inference

A residual connection adds a block's input to the output of a transformation applied to that input.

How it is used
Frames residual connections as one contributor to transformer sensitivity during weight quantization.
How deep it goes
Describes the x + sublayer(x) form and error accumulation across layers, without detailing residual-block implementation.
Examples
Illustrates the operation as x + sublayer(x) and links it to quantization errors accumulating across layers.
What you can do after
Understand why quantization errors may accumulate through layers with residual additions.

Read how Quantization and Fast Inference explains residual connection →

Also covered in

What to know first

Related concepts

Questions about residual connection

What is residual connection?

A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations so it can be combined with a later computation.

Which Manning books explain residual connection?

Inside Deep Learning: Math, Algorithms, Models, Sutskever's List, Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Deep Learning with Python, Third Edition, CUDA for LLMs, Deep Learning with PyTorch, Second Edition and 10 more.

What should I know before learning residual connection?

Manning books that teach residual connection usually introduce attention, convolutional layer, linear layer and embedding first.

What is residual connection part of?

Residual connection is part of transformer block. A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.

What is residual connection also called?

Residual connection is also called skip connection or shortcut connections.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.