Residual connection
A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
Inside Deep Learning: Math, Algorithms, Models
A bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.
- How it is used
- Uses bypass paths to make deep fully connected and convolutional networks easier to optimize.
- How deep it goes
- Explains concatenation, gradient paths, PyTorch modules, and shape constraints, then defers detailed residual variants to related treatments.
- Examples
- Demonstrates dense fully connected skips, SkipConv2d blocks, residual addition, and U-Net concatenation.
- What you can do after
- Build it from scratch with PyTorch ModuleList, torch.cat, Conv2d, BatchNorm, and shape-compatible shortcut paths.
Read how Inside Deep Learning: Math, Algorithms, Models explains residual connection →
Sutskever's List
A residual connection is a shortcut path that carries a block's input directly to an addition with the block's transformed output.
- How it is used
- Addresses the optimization difficulty that arose when greater network depth increased training error.
- How deep it goes
- Explains forward and backward paths, compares plain networks with ResNets, and examines pre-activation design through ablations.
- Examples
- Uses a telephone-game shortcut, CIFAR-10 depth comparisons, and tests that alter the identity path.
- What you can do after
- Build it from scratch as a two-path block, or diagnose whether activation and skip-path changes obstruct signal flow.
Build a Text-to-Image Generator (from Scratch)
A direct link that transfers representations between corresponding layers or stages of a neural network.
- How it is used
- Preserves information across transformed layers in transformers and ResNet50, and across matching encoder–decoder stages in denoising U-Nets.
- How deep it goes
- Defines additive residual connections and shows two implemented transformer additions; describes U-Net feature merging without specifying its merge operation.
- Examples
- Shows a transformer encoder block with two additions, ResNet50 bottleneck blocks, and a denoising U-Net with matching encoder–decoder links.
- What you can do after
- Build it from scratch in a transformer encoder block by adding each attention or feed-forward output to its sublayer input.
Read how Build a Text-to-Image Generator (from Scratch) explains residual connection →
Learn Generative AI with PyTorch
A neural-network connection that bypasses intermediate layers and passes earlier activations to later layers.
- How it is used
- Uses residual additions in deep image generators and Transformer blocks to carry inputs past transformations and support stable training.
- How deep it goes
- Defines x + f(x), traces a convolutional block, and shows two decoder-block additions without deriving gradient behavior.
- Examples
- Shows nine residual blocks in a CycleGAN generator and two residual additions per GPT-2XL decoder block.
- What you can do after
- Build it from scratch in a PyTorch residual block or decoder block by adding each sublayer’s input to its transformed output.
Read how Learn Generative AI with PyTorch explains residual connection →
Deep Learning with Python, Third Edition
- How it is used
- Keeps information and error gradients flowing through deep ConvNet blocks so deeper feature hierarchies can train.
- How deep it goes
- Works through additive Keras blocks and shape-matching projections, including downsampling, without deriving gradient equations.
- Examples
- Uses additive pseudocode, Keras ConvNet blocks, a dogs-versus-cats classifier, and a U-Net denoising autoencoder.
- What you can do after
- Apply it via the Keras Functional API by adding shortcuts around blocks and projecting them when filters or spatial dimensions change.
Read how Deep Learning with Python, Third Edition explains residual connection →
Deep Learning with PyTorch, Second Edition
A connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
- How it is used
- Provides shortcut paths that help deep convolutional networks and transformer decoders resist vanishing gradients.
- How deep it goes
- Explains the gradient problem and demonstrates addition in PyTorch residual blocks without deriving the backpropagation equations.
- Examples
- Compares convolutional networks with and without a skip connection, then shows a PyTorch residual block and a decoder diagram.
- What you can do after
- Build it from scratch in PyTorch by adding a block’s input to its transformed output.
Read how Deep Learning with PyTorch, Second Edition explains residual connection →
Grokking AI Algorithms, Second Edition
A skip connection is a direct connection that carries an earlier layer's representation to a later corresponding layer, bypassing intervening layers.
- How it is used
- Uses shortcuts to preserve a Transformer block’s input or return spatial detail to a U-Net decoder.
- How deep it goes
- Works through matching vector sizes and residual addition, then traces U-Net concatenation with tensor shapes.
- Examples
- Traces an 8D-to-4D projection and an 8×8×8 feature map concatenated with a 16×8×8 upsampled map.
- What you can do after
- Build it from scratch by adding matching Transformer vectors or concatenating U-Net feature maps before convolution.
Read how Grokking AI Algorithms, Second Edition explains residual connection →
Math and Architectures of Deep Learning
A connection that adds a layer's input to its output so information and gradients can bypass one or more layers.
- How it is used
- Explains how ResNet trains deeper convolutional networks when simply stacking layers causes degradation.
- How deep it goes
- Defines residual learning, explains identity shortcuts, and reaches PyTorch block and architecture implementations.
- Examples
- Uses CIFAR-10 depth comparisons, residual-block diagrams, PyTorch BasicBlock code, and ResNet-34 construction.
- What you can do after
- Builds residual blocks in PyTorch and assembles or invokes a ResNet-34 model.
Read how Math and Architectures of Deep Learning explains residual connection →
Deep Learning with R, Third Edition
A residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.
- How it is used
- Improves deep convolutional-network trainability by preserving representations and allowing gradients to bypass noisy processing blocks.
- How deep it goes
- Explains the shortcut mechanism, shape-matching projections, pooling case, and a reusable stacked-block implementation.
- Examples
- Uses R/Keras pseudocode, channel-changing blocks, pooling blocks, and a three-block convolutional model.
- What you can do after
- Build residual convolutional blocks with shape-compatible shortcuts and stack them into deeper image-classification models.
Read how Deep Learning with R, Third Edition explains residual connection →
Graph Neural Networks in Action
A skip connection is a neural-network pathway that passes information around one or more intermediate layers to preserve or combine representations.
- How it is used
- Uses skip connections to preserve long-range information in graph neural networks without relying solely on deeper message passing.
- How deep it goes
- Explains the motivation and trade-off, but stops short of specifying a layer-level implementation.
- Examples
- Illustrates the need with large molecular graphs, distant atoms, and toxicity-related properties.
- What you can do after
- Evaluate/understand when bypass pathways can reduce the need for excessive GNN depth.
Read how Graph Neural Networks in Action explains residual connection →
Knowledge Graphs and LLMs in Action
A residual connection is a neural-network connection that combines a layer's input with its transformed output to facilitate information and gradient propagation.
- How it is used
- Improves information and gradient flow across graph neural-network message-passing layers.
- How deep it goes
- Explains shortcut motivation and a concatenation-based update, while deferring detailed residual variants and broader architecture design.
- Examples
- Uses a GraphSAGE update function that concatenates node and aggregated-neighbor features before applying a linear transformation and ReLU.
- What you can do after
- Apply residual-style updates in GNNs to preserve features and support deeper message-passing models.
Read how Knowledge Graphs and LLMs in Action explains residual connection →
Transformers in Action
A residual connection adds a layer's input to its transformed output to facilitate optimization of deep neural networks.
- How it is used
- Supports optimization and information preservation within stacked transformer encoder sublayers.
- How deep it goes
- Explains the addition, normalization order, dropout placement, and encoder-layer use, but stops short of optimization mathematics.
- Examples
- Demonstrates residual additions around attention and feed-forward outputs in a simplified encoder-layer listing.
- What you can do after
- Build it from scratch by adding dropout-regularized sublayer outputs to their inputs before applying layer normalization.
Read how Transformers in Action explains residual connection →
Related concepts
- U-Netuses · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
- ResNetused by · ResNet is a family of deep neural-network architectures built from residual blocks with shortcut connections that facilitate information flow and optimization.
- transformer blockpart of · A transformer block is a neural-network module that transforms sequence representations using attention and feed-forward computation.
- over-smoothingaddresses · Over-smoothing is a graph neural network failure mode in which repeated neighborhood aggregation or message passing makes node representations increasingly similar and difficult to distinguish.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.