Residual connection
A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations so it can be combined with a later computation.
Inside Deep Learning: Math, Algorithms, Models
A bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.
- How it is used
- Provides a short route around learned layers to make deeper fully connected and convolutional networks easier to train.
- How deep it goes
- Derives additive paths and shape constraints, then implements same-channel blocks and channel-changing bottlenecks as PyTorch modules.
- Examples
- Uses PyTorch residual blocks and bottlenecks, a Fashion-MNIST CNN comparison, and an exercise varying residual depth.
- What you can do after
- Build it from scratch as PyTorch residual blocks and bottlenecks, choosing an identity or channel-adjusting shortcut.
Read how Inside Deep Learning: Math, Algorithms, Models explains residual connection →
Sutskever's List
A residual connection is a shortcut path that carries a block's input directly to an addition with the block's transformed output.
- How it is used
- Addresses the optimization difficulty that arose when greater network depth increased training error.
- How deep it goes
- Explains forward and backward paths, compares plain networks with ResNets, and examines pre-activation design through ablations.
- Examples
- Uses a telephone-game shortcut, CIFAR-10 depth comparisons, and tests that alter the identity path.
- What you can do after
- Build it from scratch as a two-path block, or diagnose whether activation and skip-path changes obstruct signal flow.
Build a Text-to-Image Generator (from Scratch)
A direct link that transfers representations between corresponding layers or stages of a neural network.
- How it is used
- Preserves information across transformed layers in transformers and ResNet50, and across matching encoder–decoder stages in denoising U-Nets.
- How deep it goes
- Defines additive residual connections and shows two implemented transformer additions; describes U-Net feature merging without specifying its merge operation.
- Examples
- Shows a transformer encoder block with two additions, ResNet50 bottleneck blocks, and a denoising U-Net with matching encoder–decoder links.
- What you can do after
- Build it from scratch in a transformer encoder block by adding each attention or feed-forward output to its sublayer input.
Read how Build a Text-to-Image Generator (from Scratch) explains residual connection →
Learn Generative AI with PyTorch
A neural-network connection that bypasses intermediate layers and passes earlier activations to later layers.
- How it is used
- Uses residual connections in image generators and Transformer sublayers to support information flow and stable training.
- How deep it goes
- Explains the input-plus-transformation operation and shows its implementation in convolutional and decoder blocks.
- Examples
- Uses a nine-block CycleGAN generator, GPT-2XL decoder, six-block music Transformer, and U-Net concatenation contrast.
- What you can do after
- Build residual additions into convolutional blocks and Transformer decoder sublayers using PyTorch.
Read how Learn Generative AI with PyTorch explains residual connection →
Build AI Drug Discovery Pipelines
A residual connection adds a layer's input representation to that layer's transformed output.
- How it is used
- Uses residual connections in an encoder-only transformer to preserve residue information as attention incorporates sequence context.
- How deep it goes
- Explains input-output addition, gradient flow, and subsequent layer normalization without a code implementation.
- Examples
- Illustrates preservation of an alanine’s identity alongside helix context in a protein-residue representation.
- What you can do after
- Evaluate and understand how residual addition supports information retention and gradient flow in a transformer encoder.
Read how Build AI Drug Discovery Pipelines explains residual connection →
Deep Learning with Python, Third Edition
- How it is used
- Uses shortcuts to support deeper ConvNets and preserve image detail in a denoising U-Net.
- How deep it goes
- Explains gradient flow and shape matching, then implements additive blocks and identifies concatenative shortcuts.
- Examples
- Demonstrates projected pooling blocks, a dogs-versus-cats classifier, and a denoising autoencoder.
- What you can do after
- Apply it via Keras by adding residual paths and projecting them when block outputs change shape.
Read how Deep Learning with Python, Third Edition explains residual connection →
CUDA for LLMs
A residual connection adds a sublayer's input to its output, providing a direct path through the network.
- How it is used
- Places residual connections around attention and feed-forward computations in stacked transformer blocks.
- How deep it goes
- Shows both additions in a block forward pass but does not detail a residual-specific CUDA kernel.
- Examples
- Uses a PyTorch block with separate attention and feed-forward residual additions.
- What you can do after
- Evaluate and understand how bypass paths combine with normalized sublayer outputs in a transformer block.
Deep Learning with PyTorch, Second Edition
A connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
- How it is used
- Uses shortcut paths to make deep convolutional networks and transformer decoders easier to train.
- How deep it goes
- Explains vanishing gradients and implements a convolutional residual block, including normalization and initialization.
- Examples
- Illustrates shortcuts with a three-convolution network, a reusable residual block, and a GPT-style decoder diagram.
- What you can do after
- Builds a simple residual block in PyTorch and identifies summed shortcuts in a transformer decoder.
Read how Deep Learning with PyTorch, Second Edition explains residual connection →
Grokking AI Algorithms, Second Edition
A skip connection is a direct connection that carries an earlier layer's representation to a later corresponding layer, bypassing intervening layers.
- How it is used
- Uses shortcuts to stabilize Transformer training and preserve spatial detail during U-Net image reconstruction.
- How deep it goes
- Explains shortcut purposes and traces addition and concatenation with vector and tensor dimensions.
- Examples
- Traces a four-dimensional Transformer residual and combines 16 and eight U-Net feature maps into a 24-channel tensor.
- What you can do after
- Evaluate and understand how shortcut combinations retain earlier information in Transformer and U-Net architectures.
Read how Grokking AI Algorithms, Second Edition explains residual connection →
Math and Architectures of Deep Learning
A connection that adds a layer's input to its output so information and gradients can bypass one or more layers.
- How it is used
- Explains how ResNet trains deeper convolutional networks when simply stacking layers causes degradation.
- How deep it goes
- Defines residual learning, explains identity shortcuts, and reaches PyTorch block and architecture implementations.
- Examples
- Uses CIFAR-10 depth comparisons, residual-block diagrams, PyTorch BasicBlock code, and ResNet-34 construction.
- What you can do after
- Builds residual blocks in PyTorch and assembles or invokes a ResNet-34 model.
Read how Math and Architectures of Deep Learning explains residual connection →
Deep Learning with R, Third Edition
A residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.
- How it is used
- Improves deep convolutional-network trainability by preserving representations and allowing gradients to bypass noisy processing blocks.
- How deep it goes
- Explains the shortcut mechanism, shape-matching projections, pooling case, and a reusable stacked-block implementation.
- Examples
- Uses R/Keras pseudocode, channel-changing blocks, pooling blocks, and a three-block convolutional model.
- What you can do after
- Build residual convolutional blocks with shape-compatible shortcuts and stack them into deeper image-classification models.
Read how Deep Learning with R, Third Edition explains residual connection →
Applied Reinforcement Learning
A neural-network connection that adds a layer's input directly to its transformed output.
- How it is used
- Preserves an input representation while an attention or feedforward sub-layer adds new information.
- How deep it goes
- Defines the addition as x + SubLayer(x), explains preserved information and direct gradient flow, and stops short of implementation details.
- Examples
- Uses Transformer encoder layers that wrap self-attention and feedforward sub-layers processing zone representations.
- What you can do after
- Evaluate/understand how adding a layer input to its output can preserve information and ease deep-network training.
Read how Applied Reinforcement Learning explains residual connection →
Building Specialized AI Systems
A residual connection adds a module's input to its output to support information and gradient flow through a neural network.
- How it is used
- Supports information and gradient flow while Transformer blocks process representations during training and fine-tuning.
- How deep it goes
- Defines the shortcut and locates it within Transformer blocks, but does not detail its numerical computation.
- Examples
- Shows residual connections in encoder and decoder stacks through the Add & LN pattern.
- What you can do after
- Understand where residual connections fit within a Transformer block and why shortcuts support model optimization.
Read how Building Specialized AI Systems explains residual connection →
Rearchitecting LLMs
A residual connection adds a module's input representation to its transformed output so information can bypass the transformation.
- How it is used
- Uses residual-inclusive input/output similarity to identify attention modules that may be removed.
- How deep it goes
- Explains hook locations and why excluding the residual can distort attention importance scores.
- Examples
- Uses Llama and Gemma decoder-layer hook points and contrasts importance measurement with versus without the residual.
- What you can do after
- Evaluate attention-module importance by comparing input and output cosine similarity with the residual included.
Graph Neural Networks in Action
A skip connection is a neural-network pathway that passes information around one or more intermediate layers to preserve or combine representations.
- How it is used
- Uses skip connections to preserve long-range information in graph neural networks without relying solely on deeper message passing.
- How deep it goes
- Explains the motivation and trade-off, but stops short of specifying a layer-level implementation.
- Examples
- Illustrates the need with large molecular graphs, distant atoms, and toxicity-related properties.
- What you can do after
- Evaluate/understand when bypass pathways can reduce the need for excessive GNN depth.
Read how Graph Neural Networks in Action explains residual connection →
Knowledge Graphs and LLMs in Action
A residual connection is a neural-network connection that combines a layer's input with its transformed output to facilitate information and gradient propagation.
- How it is used
- Improves information and gradient flow across graph neural-network message-passing layers.
- How deep it goes
- Explains shortcut motivation and a concatenation-based update, while deferring detailed residual variants and broader architecture design.
- Examples
- Uses a GraphSAGE update function that concatenates node and aggregated-neighbor features before applying a linear transformation and ReLU.
- What you can do after
- Apply residual-style updates in GNNs to preserve features and support deeper message-passing models.
Read how Knowledge Graphs and LLMs in Action explains residual connection →
Transformers in Action
A residual connection adds a layer's input to its transformed output to facilitate optimization of deep neural networks.
- How it is used
- Supports optimization and information preservation within stacked transformer encoder sublayers.
- How deep it goes
- Explains the addition, normalization order, dropout placement, and encoder-layer use, but stops short of optimization mathematics.
- Examples
- Demonstrates residual additions around attention and feed-forward outputs in a simplified encoder-layer listing.
- What you can do after
- Build it from scratch by adding dropout-regularized sublayer outputs to their inputs before applying layer normalization.
Read how Transformers in Action explains residual connection →
Quantization and Fast Inference
A residual connection adds a block's input to the output of a transformation applied to that input.
- How it is used
- Frames residual connections as one contributor to transformer sensitivity during weight quantization.
- How deep it goes
- Describes the x + sublayer(x) form and error accumulation across layers, without detailing residual-block implementation.
- Examples
- Illustrates the operation as x + sublayer(x) and links it to quantization errors accumulating across layers.
- What you can do after
- Understand why quantization errors may accumulate through layers with residual additions.
Read how Quantization and Fast Inference explains residual connection →
Also covered in
- Time Series Forecasting Using Foundation Models
- Build a Large Language Model (From Scratch)
- Hugging Face in Action
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Machines that ThinkIn early access (MEAP)
- Machine Learning Algorithms in Depth
- Transfer Learning for Natural Language Processing
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- convolutional layerA convolutional layer is a neural-network layer that applies shared learned filters to local regions of structured input to produce feature representations.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
Related concepts
- U-Netuses · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.
- ResNetused by · ResNet is a family of deep neural network architectures built around residual connections that help information and gradients propagate through the network.
- transformer blockpart of · A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
- ablation studyused by · An ablation study systematically removes, disables, or varies selected components or inputs and measures the resulting effects to assess their contributions to a system’s behavior or performance.
Questions about residual connection
What is residual connection?
A residual connection is a neural-network pathway that carries an earlier representation around one or more transformations so it can be combined with a later computation.
Which Manning books explain residual connection?
Inside Deep Learning: Math, Algorithms, Models, Sutskever's List, Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Deep Learning with Python, Third Edition, CUDA for LLMs, Deep Learning with PyTorch, Second Edition and 10 more.
What should I know before learning residual connection?
Manning books that teach residual connection usually introduce attention, convolutional layer, linear layer and embedding first.
What is residual connection part of?
Residual connection is part of transformer block. A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
What is residual connection also called?
Residual connection is also called skip connection or shortcut connections.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

















