The vanishing gradient problem occurs when backpropagated loss components become extremely small across successive layers, limiting effective parameter updates in deep networks. The same failure appears in GANs when a strong discriminator sends too little useful loss back to the generator. section 1.3.1 section 9.2.1
Inspect the problem through these practical signals and responses:
- Distinguish vanishing gradients from exploding gradients: successive layers can shrink gradient components toward zero or magnify them to exceptionally large values. section 1.3.1
- Use normalization between layers to keep feature-map values centered around zero and reduce both vanishing and exploding gradients; account for added computation and possible feature smoothing. section 1.3.1 section 8.1.4
- In GAN training, watch for discriminator fake loss staying within a small range while generator loss changes little and generated output fails to improve. section 9.2.2
Reproducing vanishing gradients in a GAN
In EDL_9_2_GAN_Optimization.ipynb, replace the discriminator optimizer with Adam(0.00000001, 0.5), run all cells, and inspect training output. A highly effective discriminator can leave generator loss with little useful improvement. Swapping generator ReLU activations for LeakyReLU(alpha=0.2) provides a further architecture experiment, although the observed results may remain limited. section 9.2.2
Listing 9.8 EDL_9_2_GAN_Optimization.ipynb: Setting the optimizer
gen_optimizer = Adam(0.0002, 0.5) #disc_optimizer = RMSprop(.00001) #1 # vanishing gradients disc_optimizer = Adam(.00000001, .5) #2
Inspect the optimizer setting that deliberately makes the discriminator extremely strong, then compare the resulting losses with the accompanying training-output figure. section 9.2.2
