Natural gradient descent

As explained in Bayesian Optimization in Action

Natural gradient descent is an optimization method that updates parameters using gradients adjusted to the geometry of the model's probability distribution.

Natural gradient descent adapts each update to the geometry of the loss. In a variational Gaussian process, this matters because variational parameters describe probability distributions, while ordinary parameters occupy Euclidean space. section 12.3

Figure 12.14 An example of how the loss to be minimized might behave very differently with respect to a regular parameter and a variational parameter. This gives rise to the need to take into account the geometry of the loss.

A loss can respond very differently to changes in a regular parameter and a variational parameter; treating both as ordinary Euclidean coordinates can misrepresent the useful step. section 12.3

Mini-batch gradient descent can eventually reduce the ELBO loss, but natural gradient descent uses the variational geometry to reach an effective minimum in fewer iterations. It updates variational parameters, while mini-batch gradient descent updates regular parameters such as length and output scales. section 12.5 section 12.3

Figure 12.15 An illustration of gradient descent, mini-batch gradient descent, and natural gradient descent within a loss “valley,” where the center of the valley gives the lowest loss. By accounting for the geometry of the loss function, natural gradient descent reaches the loss minimum more quickly than mini-batch.

Within a narrow loss valley, geometry-aware steps reach the center faster than ordinary or mini-batch gradient steps. section 12.3

A practical GPyTorch training setup separates the two parameter groups:

• Construct the variational distribution with NaturalVariationalDistribution and place it in a VariationalStrategy. section 12.3
• Create an NGD optimizer for model.variational_parameters(), supplying the dataset size and learning rate. section 12.3
• Create an Adam optimizer for model and likelihood parameters, compute the negative VariationalELBO, backpropagate, and step both optimizers. section 12.3
• Switch the model and likelihood to evaluation mode after training. section 12.3

Housing-price variational Gaussian process

Build a variational Gaussian process with Matérn 5/2 covariance, ARD, 100 inducing points, and a likelihood whose noise is at least 0.1. Train it for 10 iterations with natural gradient descent and compare its duration with an ordinarily trained Gaussian process using batches of 100. section 12.4

How Bayesian Optimization in Action covers it

How deep it goes

Explains Euclidean-versus-distribution geometry, optimizer separation, training steps, and faster convergence without deriving the natural-gradient formula.

Examples

Uses loss-valley illustrations, GPyTorch training code, and a housing-price exercise comparing Gaussian-process training times.

What it assumes

Assumes Gaussian-process modeling, Matérn kernels, ARD, variational models, and mini-batch training; reinforces their use through an exercise.

What you can do after

Apply it via GPyTorch to train variational Gaussian processes with separate natural-gradient and Adam optimizers.

How it connects

Connects variational Gaussian processes, inducing points, mini-batch gradient descent, Adam, and the ELBO.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• variational parameteruses · A variational parameter is a parameter that specifies a variational distribution or another component of a variational approximation.
• NaturalVariationalDistributionused by · NaturalVariationalDistribution is a GPyTorch variational-distribution implementation whose parameters are represented for natural-gradient optimization.
• gradient descentkind of · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• variational Gaussian processuses · A variational Gaussian process is a scalable approximation to a Gaussian process whose posterior is represented through variational inference over a finite set of inducing points.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.