1 Introduction to Bayesian optimization

I’m very happy that you are reading this book and excited for your upcoming journey. On a high level, Bayesian optimization is an optimization technique that may be applied when the function we are trying to optimize is a black box and expensive to evaluate, which encompasses many important tasks including hyperparameter tuning. Using Bayesian optimization could accelerate this search procedure and help us locate the optimum of the function as quickly as possible.

As a machine learning practitioner, you might have heard of the term Bayesian optimization from time to time, or you might never encounter it before. While Bayesian optimization has enjoyed enduring interest from the machine learning (ML) research community, it’s not as commonly used and talked about as other ML topics in practice. Why? Some might say Bayesian optimization has a steep learning curve: you need to understand calculus, use some probability, and overall be an experienced ML researcher to use Bayesian optimization in your application. Our goal for this book is to dispel the message that Bayesian optimization is difficult to use, and show that the technology is more accessible than one would think.

1.1 Finding the optimum of an expensive, black-box function is a difficult problem

1.1.1 Hyperparameter tuning as an example of an expensive black-box optimization problem

1.1.2 The problem of expensive, black-box optimization

1.1.3 Other real-world examples of expensive black-box optimization problems

1.2 Introducing Bayesian optimization

1.2.1 Modeling with a Gaussian process

1.2.2 Making decisions with a Bayesian optimization policy

1.2.3 Combining the Gaussian process and the optimization policy to form the optimization loop

1.2.4 Bayesian optimization in action

1.3 What will you learn in this book?

1.4 Summary