Evidence lower bound

As explained in Bayesian Optimization in Action

The evidence lower bound is a tractable variational objective that lower-bounds the marginal log likelihood and is optimized in approximate Bayesian inference.

The evidence lower bound (ELBO) is an easily computed objective for training a variational Gaussian process (VGP) when the marginal log likelihood and KL divergence are difficult to compute. It equals the marginal log likelihood minus the KL divergence, so maximizing it indirectly maximizes the marginal log likelihood while encouraging the approximate posterior to resemble the full-data posterior. section 12.2.1 section 12.5

A practical ELBO training workflow is:

• Represent the large dataset with inducing points and optimize their locations so the VGP approximates a GP trained on all data. section 12.2.1 section 12.2.3
• Estimate the loss on small data batches, whose repeated updates provide a stable approximation to the loss over the full dataset. section 12.2.2
• Use mini-batch gradient descent for regular parameters and natural gradient descent when optimizing variational parameters. section 12.5

Training a VGP with GPyTorch

In GPyTorch, construct a VariationalELBO with the likelihood, model, and total data count. For each batch, compute output = model(x_batch), set loss = -mll(output, y_batch), backpropagate, and update the model and likelihood parameters. After training, switch both to evaluation mode. The negative sign converts ELBO maximization into loss minimization. section 12.2.3

Figure 12.8 The KL divergence is decomposed into the difference between the marginal log likelihood and the evidence lower bound (ELBO). The ELBO is easy to compute and, therefore, chosen as the metric to be optimized.

The decomposition diagram connects the KL divergence, marginal log likelihood, and ELBO, showing why the ELBO is selected as the optimization metric. section 12.2.1

How Bayesian Optimization in Action covers it

How deep it goes

Defines the KL-based decomposition, explains batch estimation, and implements VGP optimization in GPyTorch.

Examples

Uses a KL-divergence decomposition, VGP inducing points, GPyTorch's VariationalELBO, and batch training code.

What it assumes

Builds on Gaussian processes, marginal log likelihood, KL divergence, inducing points, and gradient-based optimization.

What you can do after

Apply it via GPyTorch to train a variational Gaussian process on medium-to-large datasets with mini-batch updates.

How it connects

Connects marginal log likelihood, KL divergence, variational Gaussian processes, inducing points, mini-batch gradient descent, and natural gradient descent.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthThe evidence lower bound is a lower bound on the log marginal likelihood that serves as an optimization objective in variational inference.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe variational objective optimized in a VAE, combining reconstruction quality with a regularization term on the latent distribution.
• 

  liveBookmanning.com
  Causal AIThe evidence lower bound is a variational objective that lower-bounds the expected log-likelihood of observed data.

Compare how 4 books explain evidence lower bound →

Related concepts

• variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• variational inferenceused by · Variational inference is an approximate inference technique that optimizes a tractable family of probability distributions to approximate a target distribution, often a posterior.
• marginal likelihooduses · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model after marginalizing over unknown parameters or latent variables according to their prior distributions.
• objective functionkind of · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.