Depth is the number of stacked layers or sequential operations in a neural network. Increasing it generally lets the network approximate more complex functions and recognize hierarchical structure, such as a person, face, and mouth in an image. section 8.5.3
Depth has two useful interpretations:
- It adds representational complexity by composing increasingly detailed features. section 8.5.3
- It lengthens the sequence of operations performed while processing an input, much like an algorithm. section 8.5.3
- It can make training harder because long chains of derivative multiplications may cause gradients to vanish or become ineffective. section 8.5.3
From NetDepth to residual depth
A three-convolutional-layer NetDepth applies convolution, ReLU, and pooling sequentially. NetRes adds a skip connection, while NetResDeep repeats a residual block 100 times to construct a much deeper network. section 8.5.3
# In[57]:
class NetResDeep(nn.Module):
def __init__(self, n_chans1=32, n_blocks=100):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.resblocks = nn.Sequential(
*(n_blocks * [ResBlock(n_chans=n_chans1)]))
self.fc1 = nn.Linear(8 * 8 * n_chans1, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.relu(self.conv1(x)), 2)
out = self.resblocks(out)
out = F.max_pool2d(out, 2)
out = out.view(-1, 8 * 8 * self.n_chans1)
out = torch.relu(self.fc1(out))
out = self.fc2(out)
return out
Inspect a PyTorch implementation that builds depth by repeating residual blocks inside nn.Sequential. section 8.5.3
