Covariance function

As explained in Bayesian Optimization in Action

A covariance function, also called a kernel, specifies the covariance between function values at pairs of inputs in a Gaussian process.

A covariance function, or kernel, tells a Gaussian process how function values at two inputs vary together. It turns similarity into correlation: similar days can inform a weather prediction, and similar houses in the same neighborhood can inform a house-price prediction. It also shapes predictive uncertainty, which makes GP models useful for decisions under uncertainty. section 2.0 section 13.0

Formally, a covariance function takes two inputs, x1 and x2, and returns the covariance of f(x1) and f(x2). At the same input, this is the variance of f(x); at different inputs, it describes their correlation. Together with the mean function, it defines a GP as a distribution over functions rather than a fixed-size vector. section 2.3 section 3.2

Kernel choice is prior knowledge about an objective function. The covariance function specifies relationships among points and how quickly and smoothly the objective changes across its domain. A poor similarity measure can make a GP compare irrelevant features and produce unhelpful predictions; an appropriate one emphasizes features that matter to the target. section 3.2 section 13.0

Important controls and kernel choices include:

• Length scale: controls output variability as inputs change. A larger length scale produces smoother predictions and lower predictive uncertainty. section 3.6
• Output scale: controls the range of function outputs. A larger output scale yields a larger output range and greater predictive uncertainty. section 3.6
• Automatic relevance determination (ARD): assigns each input dimension its own length scale when dimensions vary at different rates. section 3.6
• RBF and Matérn kernels: the Matérn class generalizes the RBF kernel and uses its nu parameter to represent different levels of smoothness. section 3.0 section 3.6

Three-point Forrester surrogate

Fit a GP to three randomly sampled points from the Forrester function between −3 and 3, then vary covariance-function components and compare the resulting predictions. The changes make kernel selection concrete: the kernel alters the modeled structure and smoothness, not merely a numerical setting. section 3.4

Figure 13.1 The covariance between houses computed by an inappropriate kernel. Because it looks only at the color of the front door, this kernel doesn’t produce appropriate covariances.

Compare an inappropriate house-similarity kernel based only on front-door color with one that accounts for relevant features such as location and living area. section 13.0

How Bayesian Optimization in Action covers it

How deep it goes

Defines pairwise covariance formally, then develops smoothness, scales, ARD, Matérn kernels, and hyperparameter optimization.

Examples

Uses weather, house prices, a three-point Forrester function, and a front-door-color kernel to contrast useful and poor similarity measures.

What it assumes

Builds from normal and multivariate normal distributions toward Gaussian processes as distributions over functions.

What you can do after

Apply it via GPyTorch to select and tune covariance functions for GP models.

How it connects

Connects Gaussian processes with mean functions, prior knowledge, predictive uncertainty, ARD, RBF kernels, and Matérn kernels.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• kernel functionequivalent to · A kernel function is a mathematical function that measures similarity between pairs of inputs, often representing an inner product in an implicit feature space.
• Gaussian processpart of · A Gaussian process is a probability distribution over functions in which every finite collection of function values has a joint Gaussian distribution.
• prior knowledgeused by · Prior knowledge is information, beliefs, skills, or assumptions available to an agent before it encounters a task or incorporates new observations.
• smoothnessconfigures · Smoothness is the property of changing gradually, so nearby inputs produce similar outputs without discontinuities or abrupt changes.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.