2 Gaussian processes as distributions over functions

Having seen what Bayesian optimization can help us do, we are now ready to embark on our journey toward mastering Bayesian optimization. As we saw in Chapter 1, a Bayesian optimization workflow consists of two main parts: a Gaussian process (GP) as a predictive, or surrogate, model, and a policy for decision-making. With a GP, we don’t obtain only point-estimates as predictions for a test data point, but instead we have an entire probability distribution representing our belief about the prediction.

On a high level, a GP, like any other machine learning model, operates under the tenet that similar data points produce similar predictions. For example, in weather forecasting, when estimating today’s temperature, a GP will look at the climatic data of days that are similar to today, either the last few days, or this exact day a year ago. Days that are in another season wouldn’t inform the GP in this prediction. Similarly, when predicting the price of a house, a GP will say that similar houses that are in the same neighborhood as the prediction target are more informative than houses in another state.

2.1 How to sell your house the Bayesian way

2.2 Modeling correlations with multivariate Gaussian distributions and Bayesian updates

2.2.1 Using multivariate Gaussian distributions to jointly model multiple variables

2.2.2 Updating multivariate Gaussian distributions

2.2.3 Modeling many variables with high-dimensional Gaussian distributions

2.3 Going from a finite to an infinite Gaussian

2.4 Implementing Gaussian processes in Python

2.4.1 Setting up the training data

2.4.2 Implementing a Gaussian process class

2.4.3 Making predictions with a Gaussian process

2.4.4 Visualizing predictions by a Gaussian process

2.4.5 Going beyond one-dimensional objective functions

2.5 Summary

2.6 Exercise