A model can fit its training examples yet behave poorly on new ones. Weight decay helps limit that overfitting by discouraging parameters from growing large. It is one form of regularization, alongside approaches such as dropout and batch normalization. section 8.5.2 section 8.8
One way to understand the name is through L2 regularization: add a small multiple of the sum of every parameter’s square to the training loss. Under SGD, the resulting update decreases each parameter by an amount proportional to its current value. The penalty applies to biases as well as weights. Its multiplier is a hyperparameter chosen before training. section 8.5.2
# In[46]:
def training_loop_l2reg(n_epochs, optimizer, model, loss_fn,
train_loader):
for epoch in range(1, n_epochs + 1):
loss_train = 0.0
for imgs, labels in train_loader:
imgs = imgs.to(device=device)
labels = labels.to(device=device)
outputs = model(imgs)
loss = loss_fn(outputs, labels)
l2_lambda = 0.001
l2_norm = sum(p.pow(2.0).sum()
for p in model.parameters()) #1
loss = loss + l2_lambda * l2_norm
optimizer.zero_grad()
loss.backward()
optimizer.step()
loss_train += loss.item()
if epoch == 1 or epoch % 10 == 0:
print('{} Epoch {}, Training loss {}'.format(
datetime.datetime.now(), epoch,
loss_train / len(train_loader)))
The training loop sums squared model parameters, multiplies that sum by 0.001, and adds it to the loss before backpropagation and the optimizer step. section 8.5.2
The aim is smoother behavior between training examples, not simply a lower training loss. L1 regularization also penalizes large parameters, but uses absolute values rather than squares and can produce sparse weights. Another practical option is AdamW, an Adam variant that includes weight decay; the SGD explanation of L2 shrinkage does not describe AdamW’s update in detail. section 5.5.3 section 8.5.2 section 15.7.2
optimizer = torch.optim.AdamW(model.parameters(), lr=0.00006)
For a segmentation model, AdamW is instantiated with the model parameters and a specified learning rate. section 15.7.2
