Label smoothing replaces one-hot targets with softer probability distributions. Instead of treating [1, 0, 0] as absolute certainty, it may use [0.9, 0.05, 0.05], discouraging overconfident predictions, overfitting, and poorly calibrated probabilities. section 10.3.1
Training a translation Transformer
During English-to-French Transformer training, label smoothing helps the model generalize beyond its training examples. With a vocabulary-sized target, the correct token receives confidence 1 − smoothing, while smoothing mass is distributed across other tokens. Padding targets receive special handling rather than contributing ordinary target probability. section 10.3.1
Listing 10.6 A class to conduct label smoothing
class LabelSmoothing(nn.Module):
def __init__(self, size, padding_idx, smoothing=0.1):
super().__init__()
self.criterion = nn.KLDivLoss(reduction='sum')
self.padding_idx = padding_idx
self.confidence = 1.0 - smoothing
self.smoothing = smoothing
self.size = size
self.true_dist = None
def forward(self, x, target):
assert x.size(1) == self.size
true_dist = x.data.clone() #1
true_dist.fill_(self.smoothing / (self.size - 2))
true_dist.scatter_(1,
target.data.unsqueeze(1), self.confidence) #2
true_dist[:, self.padding_idx] = 0
mask = torch.nonzero(target.data == self.padding_idx)
if mask.dim() > 0:
true_dist.index_fill_(0, mask.squeeze(), 0.0)
self.true_dist = true_dist
output = self.criterion(x, true_dist.clone().detach()) #3
return output
Inspect the PyTorch implementation to see how smoothing, target-token confidence, padding indices, and KL-divergence loss work together. section 10.3.1
