Kullback–Leibler divergence measures distributional loss between an input and a learned representation. In a variational autoencoder (VAE), it complements reconstruction loss so training captures input-distribution structure rather than only pixel differences. section 8.5.2
A practical VAE loss combines:
- A reconstruction term using mean squared error or binary cross entropy. section 8.5.2
- A KL term computed from the latent mean and log variance, summed across latent dimensions and scaled by −0.5. section 8.5.2
- The mean of both terms, registered as the model loss with Keras add_loss. section 8.5.2
Listing 8.27 EDL_8_5_VAE.ipynb: Building the VAE model
use_mse = False
if use_mse:
reconstruction_loss = mse(
K.flatten(inputs), K.flatten(outputs)) #1
else:
reconstruction_loss = binary_crossentropy(
K.flatten(inputs), K.flatten(outputs)) #1
reconstruction_loss *= image_size * image_size #2
kl_loss = 1 + z_log_var - K.square(z_mean) –
K.exp(z_log_var) #3
kl_loss = K.sum(kl_loss, axis=-1) #3
kl_loss *= -0.5 #3
vae_loss = K.mean(tf.math.add(reconstruction_loss,
kl_loss)) #4
vae.add_loss(vae_loss) #5
vae.compile(optimizer='adam')
vae.summary()
Inspect the executable Keras implementation of reconstruction loss, KL divergence, combined loss, and model compilation. section 8.5.2
Comparing loss choices
Normalize the observed loss, then switch reconstruction_loss between MSE and binary cross entropy to compare training behavior despite different output scales. section 8.5.2
