Expressive power is a model’s ability to represent complicated input/output relationships, fit varied functions, or memorize training patterns. Neural networks gain expressive power by composing simple building blocks and adjusting their parameters. section 6.1.6 section 14.5.1
Width and depth are practical ways to increase expressive power. Wider layers provide more channels or features; deeper networks compose transformations across layers. More parameters can capture greater input variability and more complex functions, but can also memorize unessential details and overfit. section 8.5.1 section 8.8
Capacity in a face-to-age model
A face-to-age model with high capacity can memorize difficult training photographs, such as identifying a particular person and predicting that person’s age. It may then fail on a new person or a relative who looks similar, because memorization has replaced useful generalization. section 14.5.1
Increasing convolutional width
A PyTorch convolutional network can expose its width as n_chans1, use that value to size later convolutional and fully connected layers, and count the resulting parameters with sum(p.numel() for p in model.parameters()). section 8.5.1
# In[42]:
class NetWidth(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3,
padding=1)
self.fc1 = nn.Linear(8 * 8 * n_chans1 // 2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
out = out.view(-1, 8 * 8 * self.n_chans1 // 2)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
Shows how changing one channel-width parameter propagates through convolutional layers, the flattened feature vector, and the first linear layer. section 8.5.1
