Wasserstein GAN

As explained in Learn Generative AI with PyTorch

A GAN variant that uses the Wasserstein distance as its training objective to improve stability and provide a more informative critic score.

Wasserstein GAN (WGAN) replaces a regular GAN’s binary cross-entropy objective with the Wasserstein-1, or Earth Mover’s, distance. The change provides smoother gradient flow and more stable training, helping mitigate mode collapse, vanishing gradients, and slow convergence. section 5.2 section 5.6

Unlike a traditional discriminator that classifies inputs as real or fake, a WGAN uses a critic that assigns scores. The critic must be 1-Lipschitz continuous; gradient penalty enforces this constraint more effectively than the original weight-clipping approach. section 5.2.1

A WGAN with gradient penalty follows this training pattern:

• Replace binary cross-entropy with Wasserstein loss for real and generated data. section 5.2.1
• Sample points along straight lines between real and generated data points. section 5.2.1
• Compute the critic’s output gradient with respect to those sampled points. section 5.2.1
• Add a penalty proportional to how far the gradient norms deviate from 1. section 5.2.1

Conditional face generation

A conditional WGAN can generate human faces with or without eyeglasses. Concatenate a 100-dimensional noise vector with a two-value one-hot label, producing a 102-dimensional generator input; the critic evaluates the resulting color image. section 5.3 section 5.3.2

Figure 5.3 WGAN with gradient penalty. The discriminator network in WGAN (which we call the critic) rates input images: it tries to assign a score of –∞ to a fake image (bottom left) and a score of ∞ to the real image (top middle). Further, an interpolated image of the real and fake images (top left) is presented to the critic, and the gradient penalty with respect to the critic’s rating on the interpolated image is added to the total loss in the training process.

Use the diagram to trace critic scores for real, fake, and interpolated images, and to see where the gradient penalty enters the total loss. section 5.2.1

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains the critic, Lipschitz requirement, interpolation, gradient calculation, and penalty, without deriving the distance mathematically.

Examples

Demonstrates conditional face generation with or without eyeglasses using labeled noise, interpolated images, and critic scores.

What it assumes

Assumes familiarity with generator-discriminator GAN training and introduces the critic, Wasserstein loss, and gradient penalty.

What you can do after

Build a conditional WGAN with gradient penalty for labeled face generation and calculate Wasserstein loss and gradient penalty.

How it connects

Connects GANs with Wasserstein distance, critics, gradient penalty, Lipschitz continuity, conditional GANs, and mode collapse.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA GAN variant that uses a Wasserstein-based objective to improve training behavior and stability.

Compare how 2 books explain Wasserstein GAN →

Related concepts

• Wasserstein distanceuses · Wasserstein distance is a distance between probability distributions defined by the minimum cost of transporting probability mass between them under a specified ground distance.
• generative adversarial networkalternative to · A generative adversarial network is a generative model with a generator and discriminator trained in competition, enabling the generator to produce samples resembling a target dataset or distribution.
• mode collapseaddresses · Mode collapse is a generative-model failure in which outputs concentrate on a narrow range of patterns and fail to represent the diversity of the target distribution.
• discriminatorcontrasts with · A discriminator is the model component of a generative adversarial network that distinguishes real data from data generated by its generator.
• regularizationuses · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• gradient penaltyused by · A gradient penalty is a regularization loss term that penalizes undesirable gradients during model optimization.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.