liveBook · Manning

Evidence lower bound

As explained in Grokking Bayes

The evidence lower bound (ELBO) is a variational inference objective that lower-bounds the log marginal likelihood and is maximized to fit an approximating distribution.

The evidence lower bound (ELBO) is the score that makes variational inference practical. Instead of calculating an inaccessible posterior directly, it evaluates how well a manageable approximating distribution q fits that posterior. Maximizing the ELBO reduces the KL divergence between q and the true posterior. section 6.15 section 6.4

Use the ELBO as both an optimization target and a progress check:

  • Estimate the ELBO using values from the unnormalized posterior, q, and samples from q; Monte Carlo weighted sums approximate the required integral. section 6.4
  • Adjust the parameters of q with gradient descent so that the ELBO increases and the approximation moves closer to the target posterior. section 6.6
  • Plot the ELBO during fitting; a flattening curve suggests that optimization has converged, while an early cutoff may require more steps. section 6.10

Fitting a Normal approximation to a Beta target

Imagine a target Beta distribution on values from 0 to 1 and a Normal-based approximator q that starts concentrated near 0. Gradient descent changes q’s mean and spread: the ELBO rises as q moves toward the target, then adjusts its spread for a closer fit. The final approximation can still miss important structure when the target is multimodal or has broader variance. section 6.6 section 6.13

import matplotlib.pyplot as plt
 
plt.plot(vi_approx.hist)
plt.show()

Plotting vi_approx.hist with Matplotlib turns the ELBO history into a direct visual convergence check. section 6.10

How Grokking Bayes covers it

How deep it goes
Derives a computable KL-based objective and explains Monte Carlo estimation, gradient updates, and convergence checks.
Examples
Uses a Beta target, a squashed Normal approximator, gradient iterations, ELBO histories, and multimodal approximation failures.
What it assumes
Assumes posterior distributions, KL divergence, gradient descent, and Monte Carlo sampling, while motivating why ELBO avoids direct posterior computation.
What you can do after
Diagnose variational optimization with PyMC fitting histories and Matplotlib plots, while recognizing approximation failures.
How it connects
Connects variational inference with posterior approximation, KL divergence, gradient descent, Monte Carlo methods, and MCMC.

Also explained in

Compare how 5 books explain evidence lower bound →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.