Wasserstein distance

As explained in Learn Generative AI with PyTorch

A distance measure used in Wasserstein GAN training to compare distributions and provide a smoother optimization signal than standard GAN losses.

Wasserstein distance, also called Earth Mover’s or Wasserstein-1 distance, measures dissimilarity between two distributions. In Wasserstein GANs (WGANs), it replaces binary cross-entropy as the loss signal, producing smoother gradients and more stable training than standard GAN objectives. section 5.2 section 5.2.1

WGAN training changes the usual GAN setup in several connected ways:

• Replaces the discriminator’s binary real-or-fake classification objective with critic scores: high for real data and low for generated data. section 5.2.1 section 13.2.2
• Requires the critic to be 1-Lipschitz continuous, meaning its gradient norms must remain at most 1 everywhere. section 5.2.1 section 5.6
• Uses gradient penalty rather than relying only on weight clipping to enforce the Lipschitz constraint more effectively. section 5.2.1 section 5.6
• Helps mitigate mode collapse, vanishing gradients, and slow convergence, especially in difficult GAN training settings. section 5.2

Gradient-penalty training

For image generation, sample points along the straight line between real and generated images. Evaluate each interpolated image with the critic, compute the critic-output gradients with respect to those points, and add a penalty proportional to how far their gradient norms deviate from 1. The same pattern applies to conditional images, where interpolated samples also carry interpolated labels. section 5.2.1

Beyond images

A MuseGAN alternates critic and generator updates while applying Wasserstein distance with gradient penalty to real and generated music pieces. The generator minimizes the negative critic rating, encouraging music that receives higher scores. section 13.2.2

Figure 5.3 WGAN with gradient penalty. The discriminator network in WGAN (which we call the critic) rates input images: it tries to assign a score of –∞ to a fake image (bottom left) and a score of ∞ to the real image (top middle). Further, an interpolated image of the real and fake images (top left) is presented to the critic, and the gradient penalty with respect to the critic’s rating on the interpolated image is added to the total loss in the training process.

Use the diagram to trace how real, fake, and interpolated inputs reach the critic and how the gradient penalty enters the total loss. section 5.2.1

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains critic scores, the 1-Lipschitz constraint, interpolation, gradient norms, and penalty construction without mathematical derivation.

Examples

Uses conditional face generation, interpolated images, and MuseGAN music pieces to demonstrate Wasserstein training.

What it assumes

Assumes familiarity with generators and discriminators while teaching critics, Wasserstein loss, interpolation, and gradient penalty.

What you can do after

Apply it via WGAN-GP training loops for conditional image or music generation and monitor critic-based loss construction.

How it connects

Connects Wasserstein distance to WGANs, critics, gradient penalty, GAN training stability, mode collapse, and conditional GANs.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Platform EngineeringWasserstein distance is a measure of dissimilarity between probability distributions based on the minimum cost of transforming one distribution into another.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA distance between probability distributions that measures the minimum transport cost required to transform one distribution into another.
• 

  liveBookmanning.com
  Machine Learning System DesignWasserstein distance is a metric between probability distributions defined by the minimum cost of transporting mass from one distribution to another.

Compare how 4 books explain Wasserstein distance →

Related concepts

• Wasserstein GANused by · A Wasserstein GAN is a generative adversarial network that uses a critic and a Wasserstein-based objective to compare generated and real-data distributions.
• WGAN-GPused by · WGAN-GP is a Wasserstein generative adversarial network variant that adds a gradient penalty to constrain the critic and stabilize training.
• population stability indexalternative to · The population stability index (PSI) is a measure of the difference between two probability distributions, often used to quantify shifts in a population or model inputs.
• probability distributionmeasures · A probability distribution is a mathematical specification assigning probabilities to the possible values or outcomes of a random variable, with total probability one.
• driftmeasures · Drift is a change over time in a system’s behavior, state, data, or configuration relative to a reference or expected baseline.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.