The evidence lower bound (ELBO) is an objective for fitting a tractable variational distribution q(z) to an intractable posterior p(z|x). With observed data x and latent variables z, maximizing the ELBO learns the parameters of q without directly computing the posterior. section 3.0 section 3.6
Jensen’s inequality gives log p(x) ≥ ELBO(q) = E_q[log p(x,z)] − E_q[log q(z)]. The first term rewards probability mass where the joint model assigns high density; the second is the entropy of q. Together they favor high-probability regions without simply collapsing the approximation onto a single estimate. Equivalently, log p(x) − ELBO(q) = KL(q(z) || p(z|x)): improving q tightens the bound when the model is fixed. Expectations under q can be estimated by sampling from q. section 3.1
A likelihood-and-prior form separates fit to the observations from divergence between the approximate posterior and the prior. This makes the ELBO useful both as a derivation tool and as a quantity to compute during iterative optimization. The direction of KL matters: minimizing KL(q || p) can concentrate q on one mode of a multimodal posterior rather than cover every mode. section 3.1
Mean-field image denoising
In an Ising model, noisy observations are used to infer binary image pixels. A fully factored q assigns a variational distribution to each pixel. Mean-field updates use neighboring pixels’ current means; the ELBO can be computed term by term while iterating the updates. section 3.3
Topic inference
For latent Dirichlet allocation, a fully factored variational distribution approximates a document’s latent topic variables. Expanding the ELBO yields terms used in coordinate-ascent updates; online variational Bayes adapts the updates to large or streaming collections. section 9.1.1
Figure 9.3 Increase in ELBO vs. the number of mean-field iterations
The plotted ELBO increase shows how the objective changes over mean-field iterations in the LDA example. section 9.1.1
