Gradient penalty is an extra term in a Wasserstein GAN (WGAN) critic’s loss. It penalizes critic gradients whose norms differ from 1 at sampled points between real and generated data. This helps stabilize training and improve generated samples. section 5.2.1 section 5.3.3
A conventional GAN uses a discriminator and binary cross-entropy loss. A WGAN instead uses a critic that rates samples and a loss based on Wasserstein distance. Its critic needs a Lipschitz constraint; gradient penalty is used in place of the original WGAN’s weight clipping to encourage that constraint. section 5.2.1
To calculate the penalty for a batch:
- Blend each real sample with a generated sample using a randomly chosen mixing value. section 5.3.3
- Rate each blend with the critic, then calculate the gradient of that rating with respect to the blended input. section 5.3.3
- Take each gradient’s norm, square its difference from 1, and average across the batch. Multiply the result by a chosen weight before adding it to the critic’s loss. section 5.3.3
Listing 5.5 Calculating gradient penalty
def GP(critic, real, fake):
B, C, H, W = real.shape
alpha=torch.rand((B,1,1,1)).repeat(1,C,H,W).to(device)
interpolated_images = real*alpha+fake*(1-alpha) #1
critic_scores = critic(interpolated_images) #2
gradient = torch.autograd.grad(
inputs=interpolated_images,
outputs=critic_scores,
grad_outputs=torch.ones_like(critic_scores),
create_graph=True,
retain_graph=True)[0] #3
gradient = gradient.view(gradient.shape[0], -1)
gradient_norm = gradient.norm(2, dim=1)
gp = torch.mean((gradient_norm - 1) ** 2) #4
return gp
The PyTorch `GP()` listing shows the interpolation, `torch.autograd.grad` call, per-sample gradient norms, and averaged squared penalty. section 5.3.3
Faces with or without eyeglasses
In a conditional GAN, the generator creates face images for a requested eyeglasses label. The critic rates real and generated inputs, including their label channels. Its batch loss combines the fake rating minus the real rating with `10 * GP(critic, real, fake)`. The penalty is part of critic training, not a replacement for the real and fake ratings. section 5.3 section 5.4.2
The same pattern applies beyond images: MuseGAN blends real and generated music, obtains a critic rating for the blend, and adds its gradient penalty to the critic’s loss. section 13.2.2 section 13.5.1
