Gradient penalty
A gradient penalty is a regularization loss term that penalizes undesirable gradients during model optimization.
Learn Generative AI with PyTorch
An added loss term that penalizes undesirable gradient behavior during GAN training to improve critic stability.
- How it is used
- Stabilizes WGAN critic training for conditional image generation and music generation.
- How deep it goes
- Works through interpolation, input gradients, norm calculation, and weighted critic loss, without a dense mathematical derivation.
- Examples
- Uses labeled faces with or without glasses and generated music to demonstrate the penalty.
- What you can do after
- Apply it using PyTorch autograd to calculate a weighted critic penalty from real and generated samples.
Read how Learn Generative AI with PyTorch explains gradient penalty →
Inside Deep Learning: Math, Algorithms, Models
A regularization term that constrains discriminator gradients during Wasserstein GAN training.
- How it is used
- Stabilizes WGAN training by reducing vanishing-gradient opportunities and helping address mode collapse.
- How deep it goes
- Implements interpolation, autograd gradient calculation, a unit-norm deviation penalty, and a weight of 10 in the discriminator loss.
- Examples
- Uses a toy-grid GAN and interpolated real and generated inputs to demonstrate the penalty.
- What you can do after
- Apply via PyTorch autograd to compute the penalty on mixed samples and include it in discriminator loss.
Read how Inside Deep Learning: Math, Algorithms, Models explains gradient penalty →
Also covered in
What to know first
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Related concepts
- WGAN-GPused by · WGAN-GP is a Wasserstein generative adversarial network variant that adds a gradient penalty to constrain the critic and stabilize training.
- Wasserstein distanceuses · Wasserstein distance is a distance between probability distributions defined by the minimum cost of transporting probability mass between them under a specified ground distance.
- Wasserstein GANuses · A Wasserstein GAN is a generative adversarial network that uses a critic and a Wasserstein-based objective to compare generated and real-data distributions.
Questions about gradient penalty
What is gradient penalty?
A gradient penalty is a regularization loss term that penalizes undesirable gradients during model optimization.
Which Manning books explain gradient penalty?
Learn Generative AI with PyTorch and Inside Deep Learning: Math, Algorithms, Models.
What should I know before learning gradient penalty?
Manning books that teach gradient penalty usually introduce backpropagation and objective function first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

