Gradient penalty

Q: How deeply does Inside Deep Learning: Math, Algorithms, Models cover gradient penalty?

Derives the interpolation and autograd computation in PyTorch, then stops short of a deeper theoretical derivation.

As explained in Inside Deep Learning: Math, Algorithms, Models

A regularization term that constrains discriminator gradients during Wasserstein GAN training.

Gradient penalty regularizes the discriminator in Wasserstein GAN training. It reduces vanishing-gradient opportunities and can make it easier for the generator to keep pace, helping address mode collapse. section 9.3.3

Implement the penalty inside the discriminator update:

Sample ε values from [0, 1], shaped for broadcasting across each data example. section 9.3.3
Interpolate between real and fake data with mixed = ε·real + (1−ε)·fake. section 9.3.3
Do not detach fake before passing it to the discriminator, because the generator must remain connected to the penalty calculation. section 9.3.3

Training checks

The same training function is applied to a toy grid and 28×28 generated digits, followed by generated-sample plots and smoothed generator/discriminator loss curves.

How Inside Deep Learning: Math, Algorithms, Models covers it

Cover of Inside Deep Learning: Math, Algorithms, Models

liveBook manning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Learn Generative AI with PyTorchAn added loss term that penalizes undesirable gradient behavior during GAN training to improve critic stability.

Compare how 2 books explain gradient penalty →

Related concepts

WGAN-GPused by · WGAN-GP is a Wasserstein generative adversarial network variant that adds a gradient penalty to constrain the critic and stabilize training.
Wasserstein distanceuses · Wasserstein distance is a distance between probability distributions defined by the minimum cost of transporting probability mass between them under a specified ground distance.
Wasserstein GANuses · A Wasserstein GAN is a generative adversarial network that uses a critic and a Wasserstein-based objective to compare generated and real-data distributions.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.