Evidence lower bound
The evidence lower bound (ELBO) is a variational objective that lower-bounds a model’s log marginal likelihood and is optimized in variational inference.
Bayesian Optimization in Action
The evidence lower bound is a tractable variational objective that lower-bounds the marginal log likelihood and is optimized in approximate Bayesian inference.
- How it is used
- Optimizes a scalable variational Gaussian process when full-data marginal likelihood computation is unstable or expensive.
- How deep it goes
- Defines the KL-based decomposition, explains batch estimation, and implements VGP optimization in GPyTorch.
- Examples
- Uses a KL-divergence decomposition, VGP inducing points, GPyTorch's VariationalELBO, and batch training code.
- What you can do after
- Apply it via GPyTorch to train a variational Gaussian process on medium-to-large datasets with mini-batch updates.
Read how Bayesian Optimization in Action explains evidence lower bound →
Machine Learning Algorithms in Depth
The evidence lower bound is a lower bound on the log marginal likelihood that serves as an optimization objective in variational inference.
- How it is used
- Serves as the objective for learning variational parameters that approximate latent-variable posteriors.
- How deep it goes
- Derives the bound using Jensen’s inequality, interprets its terms, and computes it in mean-field applications.
- Examples
- Uses binary-image denoising and latent Dirichlet allocation, including an ELBO-versus-iteration plot.
- What you can do after
- Build it from scratch to guide mean-field updates and track optimization in image denoising or topic inference.
Read how Machine Learning Algorithms in Depth explains evidence lower bound →
Grokking Bayes
The evidence lower bound (ELBO) is a variational inference objective that lower-bounds the log marginal likelihood and is maximized to fit an approximating distribution.
- How it is used
- Guides variational inference by scoring approximations and monitoring gradient-based optimization.
- How deep it goes
- Derives a computable KL-based objective and explains Monte Carlo estimation, gradient updates, and convergence checks.
- Examples
- Uses a Beta target, a squashed Normal approximator, gradient iterations, ELBO histories, and multimodal approximation failures.
- What you can do after
- Diagnose variational optimization with PyMC fitting histories and Matplotlib plots, while recognizing approximation failures.
Math and Architectures of Deep Learning
The variational objective optimized in a VAE, combining reconstruction quality with a regularization term on the latent distribution.
- How it is used
- Optimizes variational autoencoders when direct posterior normalization is computationally intractable.
- How deep it goes
- Derives the lower-bound identity, interprets entropy and expected log joint, and reaches the PyTorch loss decomposition.
- Examples
- Uses MNIST reconstructions and two-dimensional latent-space plots to contrast autoencoder and VAE objectives.
- What you can do after
- Apply it via PyTorch by combining binary cross-entropy reconstruction loss with KL regularization.
Read how Math and Architectures of Deep Learning explains evidence lower bound →
Causal AI
The evidence lower bound is a variational objective that lower-bounds the expected log-likelihood of observed data.
- How it is used
- Optimizes variational autoencoder training by balancing reconstruction quality with posterior approximation.
- How deep it goes
- Explains indirect optimization of reconstruction error and KL divergence, then connects the objective to batch-based training and implementation.
- Examples
- Uses Pyro SVI configuration with Adam and Trace_ELBO, plus batch training and test-loss evaluation.
- What you can do after
- Builds VAE training with Pyro’s SVI and Trace_ELBO while monitoring training and test loss.
What to know first
- prior probabilityA prior probability or distribution is a probability or probability distribution assigned to an event, hypothesis, or unknown quantity before relevant evidence is incorporated.
- adamAdam is an adaptive stochastic-gradient optimization algorithm that updates parameters using exponential moving estimates of the first and second moments of gradients.
- approximate distributionAn approximate posterior distribution is a tractable probability distribution chosen or optimized to approximate a target posterior distribution over unknown quantities.
- marginal likelihoodMarginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model, averaged over its unknown parameters or latent variables according to their prior distributions.
Related concepts
- variational inferenceused by · Variational inference is an approximate inference technique that optimizes a tractable family of probability distributions to approximate a target distribution, often a posterior.
- variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
- marginal likelihooduses · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model, averaged over its unknown parameters or latent variables according to their prior distributions.
- objective functionkind of · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- Kullback–Leibler divergencealternative to · Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.
Questions about evidence lower bound
What is evidence lower bound?
The evidence lower bound (ELBO) is a variational objective that lower-bounds a model’s log marginal likelihood and is optimized in variational inference.
Which Manning books explain evidence lower bound?
Bayesian Optimization in Action, Machine Learning Algorithms in Depth, Grokking Bayes, Math and Architectures of Deep Learning and Causal AI.
What should I know before learning evidence lower bound?
Manning books that teach evidence lower bound usually introduce prior probability, adam, approximate distribution and marginal likelihood first.
What is the difference between evidence lower bound and Kullback–Leibler divergence?
The evidence lower bound (ELBO) is a variational objective that lower-bounds a model’s log marginal likelihood and is optimized in variational inference. Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.
What is evidence lower bound a kind of?
Evidence lower bound is a kind of objective function. An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
What is evidence lower bound also called?
Evidence lower bound is also called ELBO.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




