A model can fit its training data yet perform poorly on examples it has never seen. Regularization is a set of techniques that helps prevent this overfitting. The goal is not just to lower training loss, but to keep behavior sensible on independent validation data. section 8.5.2 section 5.5.3
One approach adds a penalty to the loss so large weights become costly. L2 regularization uses the sum of squared weights; L1 uses the sum of their absolute values and can produce sparse weights. A small hyperparameter controls the penalty’s strength. L2 is also called weight decay because its contribution to an SGD update reduces each weight in proportion to its current value. section 8.5.2
Add an L2 penalty in PyTorch
After computing the usual loss, add `l2_lambda * sum(p.pow(2).sum() for p in model.parameters())`, then call `optimizer.zero_grad()`, `loss.backward()`, and `optimizer.step()`. The worked training loop uses `l2_lambda = 0.001`; that value is a chosen hyperparameter, not part of the definition of L2. section 8.5.2
# In[46]:
def training_loop_l2reg(n_epochs, optimizer, model, loss_fn,
train_loader):
for epoch in range(1, n_epochs + 1):
loss_train = 0.0
for imgs, labels in train_loader:
imgs = imgs.to(device=device)
labels = labels.to(device=device)
outputs = model(imgs)
loss = loss_fn(outputs, labels)
l2_lambda = 0.001
l2_norm = sum(p.pow(2.0).sum()
for p in model.parameters()) #1
loss = loss + l2_lambda * l2_norm
optimizer.zero_grad()
loss.backward()
optimizer.step()
loss_train += loss.item()
if epoch == 1 or epoch % 10 == 0:
print('{} Epoch {}, Training loss {}'.format(
datetime.datetime.now(), epoch,
loss_train / len(train_loader)))
The training loop shows exactly where the L2 penalty enters the loss before backpropagation. section 8.5.2
Other ways to limit overfitting or support generalization include:
- Reduce model capacity if a larger model fits training data but overfits. It still needs enough capacity to learn the task. section 5.5.3 section 8.8
- Add noise to input samples as data augmentation, creating examples that discourage memorization of exact training points. section 5.5.3
- Use dropout or batch normalization in a convolutional network; both appear alongside weight decay as regularization strategies. section 8.5.2 section 8.8
# In[48]:
class NetDropout(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv1_dropout = nn.Dropout2d(p=0.4)
self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3,
padding=1)
self.conv2_dropout = nn.Dropout2d(p=0.4)
self.fc1 = nn.Linear(8 * 8 * n_chans1 // 2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
out = self.conv1_dropout(out)
out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
out = self.conv2_dropout(out)
out = out.view(-1, 8 * 8 * self.n_chans1 // 2)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
This convolutional network places `nn.Dropout2d(p=0.4)` after each of two pooled convolutional outputs. section 8.5.2
# In[50]:
class NetBatchNorm(nn.Module):
def __init__(self, n_chans1=32):
super().__init__()
self.n_chans1 = n_chans1
self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)
self.conv1_batchnorm = nn.BatchNorm2d(num_features=n_chans1)
self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3,
padding=1)
self.conv2_batchnorm = nn.BatchNorm2d(num_features=n_chans1 // 2)
self.fc1 = nn.Linear(8 * 8 * n_chans1 // 2, 32)
self.fc2 = nn.Linear(32, 2)
def forward(self, x):
out = self.conv1_batchnorm(self.conv1(x))
out = F.max_pool2d(torch.tanh(out), 2)
out = self.conv2_batchnorm(self.conv2(out))
out = F.max_pool2d(torch.tanh(out), 2)
out = out.view(-1, 8 * 8 * self.n_chans1 // 2)
out = torch.tanh(self.fc1(out))
out = self.fc2(out)
return out
This alternative places `nn.BatchNorm2d` after each convolution; the accompanying figure illustrates normalization using feature means and standard deviations. section 8.5.2
