A residual connection adds a module’s input to its output. This shortcut helps information and gradients flow through repeated Transformer blocks, supporting training and fine-tuning. section 3.1.1
Within a Transformer block, residual connections appear alongside:
- Self-attention and position-wise feed-forward networks, which transform token representations. section 3.1.1
- Normalization, forming the Add & LN pattern shown in encoder and decoder stacks. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

Inspect the Transformer architecture to see residual connections paired with layer normalization in encoder and decoder blocks. section 3.1.1

















