Gaussian process

As explained in Bayesian Optimization in Action

A Gaussian process is a probabilistic model over functions that defines predictive distributions, including means and uncertainties, at unobserved inputs.

A Gaussian process (GP) is a probabilistic model of a function: at any finite set of inputs, its function values have a multivariate normal distribution. Rather than returning only one predicted value, it returns a predictive distribution, including a mean and uncertainty. Similar inputs are expected to produce similar predictions; the covariance function encodes that similarity and uncertainty. section 2.3 section 2.0

A GP can represent beliefs before observations and update them after data arrives. A prior GP can have a constant mean and confidence interval while sampled functions still show complex nonlinear shapes. After conditioning on observations, predictions smoothly interpolate nearby data and uncertainty shrinks around those observations while remaining larger farther away. section 2.4.4 section 2.6

In Bayesian optimization, a GP serves as the surrogate model for an expensive black-box objective:

• Train the GP on observed input–output pairs to form a belief about the objective across the search domain. section 1.2.3
• Provide mean predictions and uncertainty to a policy, which scores candidate points by their value for locating the global optimum. section 1.2.3
• Update the training data with the selected evaluation and repeat until the evaluation budget is exhausted. section 1.2.3

Exploration after a local optimum

On a one-dimensional objective, early evaluations can cluster near a local optimum. Once that region has been explored sufficiently, the GP-informed Expected Improvement policy shifts attention toward uncertain, promising regions near other parts of the domain, eventually locating the global optimum. section 1.2.4

A practical GPyTorch implementation defines mean and covariance modules and returns a multivariate normal distribution. With a Gaussian likelihood, prediction mode produces a predictive distribution whose mean, confidence region, and samples can be plotted. Training commonly minimizes negative marginal log likelihood with gradient descent; regular GPs can become slow and numerically unstable as datasets grow. section 2.4.3 section 12.1.2 section 12.1.3

Figure 2.18 Predictions made by a posterior GP. The mean function and randomly drawn samples smoothly interpolate the training data points, while uncertainty vanishes in the regions surrounding these data points.

Compare posterior mean, sampled functions, and uncertainty around training points to see how observations constrain a GP. section 2.4.4

How Bayesian Optimization in Action covers it

How deep it goes

Defines GPs through finite multivariate-normal subsets, explains prior-to-posterior behavior, and implements exact models; later scaling issues are identified.

Examples

Uses housing-price regions, prior and posterior function plots, and one-dimensional black-box optimization iterations.

What it assumes

Builds from a crash course in multivariate Gaussian distributions and introduces GPs as distributions over functions.

What you can do after

Build exact GPyTorch GP models, visualize predictive distributions, train them with gradient descent, and use them within BayesOpt.

How it connects

Connects multivariate Gaussian distributions, covariance functions, Bayesian updating, Gaussian likelihoods, and exploration–exploitation policies.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  AI-Powered SearchA Gaussian process is a collection of random variables whose finite subsets have joint Gaussian distributions and that defines a probability distribution over functions.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA Gaussian process is a collection of random variables such that every finite subset has a jointly Gaussian distribution, defining a probability distribution over functions.

Compare how 3 books explain Gaussian process →

Related concepts

• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• covariance functionhas part · A covariance function, also called a kernel, specifies the covariance between function values at pairs of inputs in a Gaussian process.
• mean functionhas part · A mean function maps an input to the expected value of a random function before conditioning on observed data.
• Bayesian optimizationused by · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• kernel functionuses · A kernel function is a mathematical function that measures similarity between pairs of inputs, often representing an inner product in an implicit feature space.
• radial basis function kerneluses · A radial basis function kernel is a similarity function whose value depends on the distance between inputs and typically decreases as that distance increases.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.