A residual connection adds a transformation’s output to its input: x + sublayer(x). In transformer quantization, errors can accumulate through layers with these additions, contributing to sensitivity. section 3.2
Residual connection
A residual connection adds a block's input to the output of a transformation applied to that input.
How Quantization and Fast Inference covers it
- How deep it goes
- Describes the x + sublayer(x) form and error accumulation across layers, without detailing residual-block implementation.
- Examples
- Illustrates the operation as x + sublayer(x) and links it to quantization errors accumulating across layers.
- What you can do after
- Understand why quantization errors may accumulate through layers with residual additions.
- How it connects
- Ties residual connections to transformer attention, LayerNorm, and quantization sensitivity.
Also explained in
-
Inside Deep Learning: Math, Algorithms, ModelsA bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.
-
Sutskever's ListA residual connection is a shortcut path that carries a block's input directly to an addition with the block's transformed output.
-
Build a Text-to-Image Generator (from Scratch)A direct link that transfers representations between corresponding layers or stages of a neural network.
-
Learn Generative AI with PyTorchA neural-network connection that bypasses intermediate layers and passes earlier activations to later layers.
-
Build AI Drug Discovery PipelinesA residual connection adds a layer's input representation to that layer's transformed output.
-
CUDA for LLMsA residual connection adds a sublayer's input to its output, providing a direct path through the network.
-
Deep Learning with PyTorch, Second EditionA connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
-
Grokking AI Algorithms, Second EditionA skip connection is a direct connection that carries an earlier layer's representation to a later corresponding layer, bypassing intervening layers.
-
Math and Architectures of Deep LearningA connection that adds a layer's input to its output so information and gradients can bypass one or more layers.
-
Deep Learning with R, Third EditionA residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.
-
Applied Reinforcement LearningA neural-network connection that adds a layer's input directly to its transformed output.
-
Building Specialized AI SystemsA residual connection adds a module's input to its output to support information and gradient flow through a neural network.
-
Rearchitecting LLMsA residual connection adds a module's input representation to its transformed output so information can bypass the transformation.
-
Graph Neural Networks in ActionA skip connection is a neural-network pathway that passes information around one or more intermediate layers to preserve or combine representations.
-
Knowledge Graphs and LLMs in ActionA residual connection is a neural-network connection that combines a layer's input with its transformed output to facilitate information and gradient propagation.
-
Transformers in ActionA residual connection adds a layer's input to its transformed output to facilitate optimization of deep neural networks.
Compare how 18 books explain residual connection →
Related concepts
- U-Netuses · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.
- ResNetused by · ResNet is a family of deep neural network architectures built around residual connections that help information and gradients propagate through the network.
- transformer blockpart of · A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.
- ablation studyused by · An ablation study systematically removes, disables, or varies selected components or inputs and measures the resulting effects to assess their contributions to a system’s behavior or performance.
This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.

















