Variational parameter

As explained in Bayesian Optimization in Action

A variational parameter is a parameter that specifies a variational distribution or another component of a variational approximation.

A variational parameter specifies part of a variational approximation in a variational Gaussian process (VGP). Unlike regular parameters such as a mean constant or covariance length scale, it is associated with probability distributions and the inducing points used to construct the approximation. Its behavior is therefore not well represented by ordinary Euclidean distance. section 12.3

Figure 12.14 An example of how the loss to be minimized might behave very differently with respect to a regular parameter and a variational parameter. This gives rise to the need to take into account the geometry of the loss.

The loss can behave very differently along a regular parameter and a variational parameter, motivating optimization methods that account for loss-surface geometry. section 12.3

Variational parameters matter during VGP training because:

• A change in an inducing point's mean is not equivalent to the same numerical change in a covariance length scale, even when both changes have equal Euclidean size. section 12.3
• Natural gradient descent uses the geometry of the loss with respect to variational parameters, while mini-batch gradient descent optimizes regular parameters such as length and output scales. section 12.5
• In GPyTorch, variational parameters can be exposed through model.variational_parameters() and updated with gpytorch.optim.NGD alongside an Adam optimizer for hyperparameters. section 12.3

How Bayesian Optimization in Action covers it

How deep it goes

Defines the parameter distinction, explains its geometric consequence, and demonstrates separate natural-gradient and hyperparameter optimization.

Examples

Contrasts length-scale and inducing-mean loss behavior and provides GPyTorch code using NaturalVariationalDistribution and NGD.

What it assumes

Assumes familiarity with Gaussian processes, VGPs, inducing points, and gradient-based training without requiring advanced distribution geometry.

What you can do after

Apply variational-parameter optimization through GPyTorch's NGD optimizer while updating regular parameters with Adam.

How it connects

Connects variational Gaussian processes with inducing points, variational distributions, natural gradient descent, mini-batch gradient descent, and ELBO optimization.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• natural gradient descentused by · Natural gradient descent is an optimization method that updates parameters using gradients adjusted to the geometry of a model’s probability distribution.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.