A residual connection passes an input around a set of transformations and adds it to their output: x + f(x). The transformed path can change the input while the direct path carries it forward. This helps address vanishing gradients in deep networks. section 6.3.2
A residual block in an image generator
In a CycleGAN generator, x passes through two convolution-and-instance-normalization stages, with a ReLU between them. The block adds the original x to the result. Nine such blocks sit between the generator’s earlier convolutions and its upsampling blocks. section 6.3.2
Figure 6.4 The architecture of a residual block. The input x is passed through a series of transformations (two sets of Conv2d layer and InstanceNorm2d layer and a ReLU activation). The input x is then added back to the output of these transformations, f(x). The output of the residual block is therefore x + f(x).
Follow the two paths through the block: one transforms x into f(x), while the other carries x directly to the final addition. section 6.3.2
The same addition appears in Transformer decoder blocks. In the GPT-2XL implementation, each block first computes x + attention(layer_norm(x)), then adds a layer-normalized feed-forward result to that updated x. Residual connections therefore surround both sublayers, rather than only the whole decoder block. section 11.2.4
Listing 11.5 Constructing a decoder block
class Block(nn.Module):
def __init__(self, config): #1
super().__init__()
self.ln_1 = nn.LayerNorm(config.n_embd)
self.attn = CausalSelfAttention(config)
self.ln_2 = nn.LayerNorm(config.n_embd)
self.mlp = nn.ModuleDict(dict(
c_fc = nn.Linear(config.n_embd, 4 * config.n_embd),
c_proj = nn.Linear(4 * config.n_embd, config.n_embd),
act = GELU(),
dropout = nn.Dropout(config.resid_pdrop),
))
m = self.mlp
self.mlpf=lambda x:m.dropout(m.c_proj(m.act(m.c_fc(x))))
def forward(self, x):
x = x + self.attn(self.ln_1(x)) #2
x = x + self.mlpf(self.ln_2(x)) #3
return x
The two additions in Block.forward show exactly where the attention and feed-forward paths rejoin their inputs. section 11.2.4
Do not confuse this addition with the skip connections in a denoising U-Net. Those connections carry encoder feature maps past the bottleneck and concatenate them with decoder feature maps to preserve details such as edges and textures. For example, two feature maps shaped (128, 64, 64) combine into a decoder input shaped (256, 64, 64). section 15.1.2
