liveBook · Manning

P(|)

As explained in Math and Architectures of Deep Learning

The true target posterior distribution over latent variables that the variational approximation tries to match.

p(|) is the unknown true posterior distribution over latent variables. Variational inference learns q(|), an approximation chosen to be as close as possible to p(|). section 14.7.6

The approximation is optimized through the relationship between KL divergence and the ELBO:

  • Minimizing KL divergence between q(|) and p(|) is equivalent to maximizing the evidence lower bound. section 14.7.6
  • The identity KLD(q, p) + ELBO = ln(p()) shows why ELBO is a lower bound: KL divergence cannot be negative. section 14.7.6
  • The entropy component rewards a diffuse q(|), encouraging smoothness in the latent space. section 14.7.6

Approximation objective

Rather than directly fitting the inaccessible p(|), optimize q(|) by maximizing ELBO = Hq + Eq(ln(p(, ))). section 14.7.6

KLD(q, p) + ELBO = constant

Use the identity connecting KL divergence, ELBO, and the evidence log-likelihood to visualize the approximation objective. section 14.7.6

How Math and Architectures of Deep Learning covers it

How deep it goes
Defines the target distribution and derives its KL-to-ELBO relationship, stopping short of specifying its density.
Examples
Uses the ELBO decomposition and the identity KLD(q, p) + ELBO = ln(p()) as concrete mathematical illustrations.
What it assumes
Assumes familiarity with probability distributions while introducing KL divergence and entropy as supporting ideas.
What you can do after
Evaluate how variational inference replaces an unknown posterior with an ELBO-optimized approximation.
How it connects
Links latent variables with variational inference, KL divergence, entropy, ELBO, and variational autoencoders.
Cover of Math and Architectures of Deep Learning
liveBookmanning.com
Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.