A residual connection lets an input bypass a transformation and adds it to the transformation’s output: output = input + transformation(input). The transformed path can learn new features while the direct path preserves information. In deep networks, this shortcut also helps combat vanishing gradients, making deeper models easier to train. section 13.3.1 section 3.3.4
In a transformer encoder block, the pattern appears twice: once around multi-head self-attention and once around the feed-forward network. The illustrated implementation normalizes the input before each transformation, then adds the original input back. Decoder sublayers use residual connections too, including the cross-attention sublayer. section 3.3.4 section 2.3.2
Listing 3.5 Creating an encoder block
class Block(nn.Module):
def __init__(self, config):
super().__init__()
self.attention = MultiHeadAttention(config)
self.layernorm_1 = nn.LayerNorm(config.hidden_size)
self.mlp = MLP(config)
self.layernorm_2 = nn.LayerNorm(config.hidden_size)
def forward(self, x, output_attentions=False):
attention_output, attention_probs = \
self.attention(self.layernorm_1(x), #1
output_attentions=output_attentions)
x = x + attention_output #2
mlp_output = self.mlp(self.layernorm_2(x)) #3
x = x + mlp_output #4
if not output_attentions:
return (x, None)
else:
return (x, attention_probs)
Follow the two `x = x + ...` lines: each adds a sublayer’s result to the input that reached that sublayer. section 3.3.4
ResNet50 applies the same additive shortcut idea inside convolutional bottleneck blocks. A related use of skip connections appears in a denoising U-Net: links between encoder and decoder stages of the same resolution merge earlier feature maps into the upsampling path. Those links restore fine detail that downsampling could otherwise lose; unlike the transformer block’s explicit addition, the cited U-Net description does not specify the merge operation. section 13.3.1 section 6.2.2 section 6.5
Skip connections are useful, but they do not select relevant features on their own. In the denoising U-Net, redundant feature extraction can make that selection harder; attention is added to emphasize important image regions while disregarding irrelevant ones. section 7.1.1
