liveBook · Manning

Elbo trick

As explained in Grokking Bayes

The ELBO trick is a variational-inference technique that applies Bayes' theorem to rewrite an intractable KL-divergence objective using a computable evidence lower bound.

Variational inference needs to compare a simple approximating distribution q with the unknown posterior p(θ | D). Directly computing their KL divergence is difficult because the posterior is unavailable. The ELBO trick applies Bayes’ theorem to rewrite that objective using an evidence lower bound that is easier to calculate. section 6.4

The calculation becomes manageable by combining three ideas:

  • Compute the unnormalized posterior inside the logarithm using p(θ | D)p(θ). section 6.4
  • Choose q(θ) to be simple enough to evaluate. section 6.4
  • Approximate the remaining integral with a weighted sum of samples drawn from q(θ), using Monte Carlo estimation. section 6.4

Use the equation to follow how Bayes’ theorem turns the KL-divergence objective into the ELBO expression. section 6.4

How Grokking Bayes covers it

How deep it goes
Explains the rewrite, computable terms, and Monte Carlo integral approximation without developing implementation details.
Examples
Uses q-samples and weighted sums as the concrete computational illustration.
What you can do after
Apply it via Monte Carlo sampling to estimate the difficult integral in the ELBO objective.
How it connects
Connects KL divergence, Bayes’ theorem, variational inference, posterior distributions, Monte Carlo methods, and MCMC.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.