Gradient penalty
A gradient penalty is a regularization loss term that penalizes undesirable gradients during model optimization.
Learn Generative AI with PyTorch
An added loss term that penalizes undesirable gradient behavior during GAN training to improve critic stability.
- How it is used
- Stabilizes WGAN critic training by adding a gradient-based regularization term to its loss.
- How deep it goes
- Defines the penalty and works through interpolation, automatic differentiation, gradient norms, and critic-loss integration without a dense derivation.
- Examples
- Uses labeled face generation and MuseGAN music generation, with real–fake blends in both.
- What you can do after
- Build it from scratch in PyTorch and add it to a WGAN critic’s batch loss.
Read how Learn Generative AI with PyTorch explains gradient penalty →
Inside Deep Learning: Math, Algorithms, Models
A regularization term that constrains discriminator gradients during Wasserstein GAN training.
- How it is used
- Applies gradient regularization during Wasserstein GAN discriminator training to reduce vanishing gradients and mode collapse.
- How deep it goes
- Derives the interpolation and autograd computation in PyTorch, then stops short of a deeper theoretical derivation.
- Examples
- Uses a toy grid, generated 28×28 digits, sample plots, and smoothed generator and discriminator loss curves.
- What you can do after
- Apply it via PyTorch by interpolating samples, differentiating discriminator outputs, and adding the weighted penalty to discriminator loss.
Read how Inside Deep Learning: Math, Algorithms, Models explains gradient penalty →
Related concepts
- WGAN-GPused by · WGAN-GP is a Wasserstein generative adversarial network variant that adds a gradient penalty to constrain the critic and stabilize training.
- Wasserstein distanceuses · Wasserstein distance is a distance between probability distributions defined by the minimum cost of transporting probability mass between them under a specified ground distance.
- Wasserstein GANuses · A Wasserstein GAN is a generative adversarial network that uses a critic and a Wasserstein-based objective to compare generated and real-data distributions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.