liveBook · Manning

Gradient penalty

A gradient penalty is a regularization loss term that penalizes undesirable gradients during model optimization.

How 2 Manning books explain it

Learn Generative AI with PyTorch

An added loss term that penalizes undesirable gradient behavior during GAN training to improve critic stability.

How it is used
Stabilizes WGAN critic training for conditional image generation and music generation.
How deep it goes
Works through interpolation, input gradients, norm calculation, and weighted critic loss, without a dense mathematical derivation.
Examples
Uses labeled faces with or without glasses and generated music to demonstrate the penalty.
What you can do after
Apply it using PyTorch autograd to calculate a weighted critic penalty from real and generated samples.

Read how Learn Generative AI with PyTorch explains gradient penalty →

Inside Deep Learning: Math, Algorithms, Models

A regularization term that constrains discriminator gradients during Wasserstein GAN training.

How it is used
Stabilizes WGAN training by reducing vanishing-gradient opportunities and helping address mode collapse.
How deep it goes
Implements interpolation, autograd gradient calculation, a unit-norm deviation penalty, and a weight of 10 in the discriminator loss.
Examples
Uses a toy-grid GAN and interpolated real and generated inputs to demonstrate the penalty.
What you can do after
Apply via PyTorch autograd to compute the penalty on mixed samples and include it in discriminator loss.

Read how Inside Deep Learning: Math, Algorithms, Models explains gradient penalty →

Also covered in

What to know first

Related concepts

Questions about gradient penalty

What is gradient penalty?

A gradient penalty is a regularization loss term that penalizes undesirable gradients during model optimization.

Which Manning books explain gradient penalty?

Learn Generative AI with PyTorch and Inside Deep Learning: Math, Algorithms, Models.

What should I know before learning gradient penalty?

Manning books that teach gradient penalty usually introduce backpropagation and objective function first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.