Variational inference needs to compare a simple approximating distribution q with the unknown posterior p(θ | D). Directly computing their KL divergence is difficult because the posterior is unavailable. The ELBO trick applies Bayes’ theorem to rewrite that objective using an evidence lower bound that is easier to calculate. section 6.4
The calculation becomes manageable by combining three ideas:
- Compute the unnormalized posterior inside the logarithm using p(θ | D)p(θ). section 6.4
- Choose q(θ) to be simple enough to evaluate. section 6.4
- Approximate the remaining integral with a weighted sum of samples drawn from q(θ), using Monte Carlo estimation. section 6.4
Use the equation to follow how Bayes’ theorem turns the KL-divergence objective into the ELBO expression. section 6.4
