A residual connection, also called a skip connection, gives a network a shortcut around a layer or block of layers. Instead of using only the transformed output, the network adds the earlier activation back in. This gives gradients a more direct path during backpropagation. section 8.5.3 section 9.6.1
Why add the shortcut? In a very deep network, gradients reaching early layers pass through a long chain of operations and can become too small for those layers to train effectively. Residual connections help combat this vanishing-gradient problem, making deeper networks practical. section 8.5.3 section 8.8
A small PyTorch residual block
A block can transform x with a convolution, batch normalization, and ReLU, then return out + x. The addition is the residual connection: x bypasses those operations. The two tensors must have compatible sizes for the addition; padding can preserve image size across convolutions. section 8.5.3 section 8.2.1
# In[56]:
class ResBlock(nn.Module):
def __init__(self, n_chans):
super(ResBlock, self).__init__()
self.conv = nn.Conv2d(n_chans, n_chans, kernel_size=3,
padding=1, bias=False) #1
self.batch_norm = nn.BatchNorm2d(num_features=n_chans)
torch.nn.init.kaiming_normal_(self.conv.weight,
nonlinearity='relu') #2
torch.nn.init.constant_(self.batch_norm.weight, 0.5)
torch.nn.init.zeros_(self.batch_norm.bias)
def forward(self, x):
out = self.conv(x)
out = self.batch_norm(out)
out = torch.relu(out)
return out + x
The return statement shows the shortcut in code: the transformed output is added to the block’s input. section 8.5.3
The same idea appears in a GPT-style transformer decoder. Its residual connections sum outputs at points in the decoder block, giving gradients paths that bypass intervening layers. section 9.6.1
Figure 9.10 GPT-style decoder. The pluses represent residual connections where outputs are summed.
The plus signs mark where the decoder sums outputs through residual connections. section 9.6.1
