liveBook · Manning

Residual connection

As explained in Building Specialized AI Systems

A residual connection adds a module's input to its output to support information and gradient flow through a neural network.

A residual connection adds a module’s input to its output. This shortcut helps information and gradients flow through repeated Transformer blocks, supporting training and fine-tuning. section 3.1.1

Within a Transformer block, residual connections appear alongside:

  • Self-attention and position-wise feed-forward networks, which transform token representations. section 3.1.1
  • Normalization, forming the Add & LN pattern shown in encoder and decoder stacks. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

Inspect the Transformer architecture to see residual connections paired with layer normalization in encoder and decoder blocks. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Defines the shortcut and locates it within Transformer blocks, but does not detail its numerical computation.
Examples
Shows residual connections in encoder and decoder stacks through the Add & LN pattern.
What you can do after
Understand where residual connections fit within a Transformer block and why shortcuts support model optimization.
How it connects
Connects residual connections with Transformer blocks, self-attention, feed-forward networks, layer normalization, and encoder-decoder architectures.

Also explained in

Compare how 18 books explain residual connection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.