ResNet builds deep convolutional networks from residual blocks. Each block combines a learned long path with a shortcut path, helping information pass through many layers. A bottleneck uses 1×1 convolutions to reduce and restore channels around a central convolution; an identity shortcut handles unchanged channel counts, while a 1×1 convolution with batch normalization handles changes. section 6.5.4
class ResidualBottleNeck(nn.Module):
def __init__(self, in_channels, out_channels,
kernel_size=3, leak_rate=0.1):
super().__init__()
pad = (kernel_size-1)//2 #1
bottleneck = max(out_channels//4, in_channels) #2
self.F = nn.Sequential(
nn.BatchNorm2d(in_channels), #4
nn.LeakyReLU(leak_rate),
nn.Conv2d(in_channels, bottleneck, 1, padding=0),
nn.BatchNorm2d(bottleneck), #5
nn.LeakyReLU(leak_rate),
nn.Conv2d(bottleneck, bottleneck, kernel_size, padding=pad),
nn.BatchNorm2d(bottleneck), #3#6
nn.LeakyReLU(leak_rate),
nn.Conv2d(bottleneck, out_channels, 1, padding=0)
)
self.shortcut = nn.Identity() #7
if in_channels != out_channels: #8
self.shortcut = nn.Sequential(
nn.Conv2d(in_channels, out_channels, 1, padding=0),
nn.BatchNorm2d(out_channels)
)
def forward(self, x):
return self.shortcut(x) + self.F(x) #9
#1 How much padding will our convolutional layers need to maintain the input shape?
#2 The bottleneck should be smaller, so output /4 or input. You could also try changing max to min; it’s not a major issue.
#3 Defines the three sets of BN and convolution layers we need. Notice that for the 1 convs, we use padding=0, because 1 will not change shape!
#4 Compresses down
#5 Normal layer doing a full conv
#6 Expands back up
#7 By default, our shortcut is the identity function, which simply returns the input as the output.
#8 If we need to change the shape, let’s turn the shortcut into a small layer with 1 conv and BM.
#9 “shortcut(x)” plays the role of “x"; we do as little work as possible to keep the tensor shapes the same.
The implementation exposes the two paths directly: self.F contains batch normalization, LeakyReLU, and convolutions, while self.shortcut is either identity or a channel-adjusting sequential module. section 6.5.4
A residual classifier can stack bottlenecks, ordinary residual blocks, activations, and pooling before flattening into a linear classifier. ResNet-18 and related ResNet-X names identify architectures by their approximate layer counts; adaptive pooling lets the architecture accept arbitrarily sized input images before classification. section 6.5.4 section 13.2
Adapting ResNet-18
Instantiate torchvision.models.resnet18(), then replace model.fc with nn.Linear(model.fc.in_features, 2) for a two-class classifier. With pretrained=True, the convolutional filters start from ImageNet training while the replacement fully connected layer starts randomly. section 13.2 section 13.2.1
