When each trial is costly, testing every possible setting can waste time. Bayesian optimization searches an expensive or time-consuming objective sequentially, using results from earlier evaluations to guide what to test next. section 2.4.1
The search proceeds by updating beliefs about the unknown relationship between candidate settings and performance:
- Begin with a prior belief about how settings relate to the objective. section 2.4.1
- Evaluate a candidate and use its observed result to update the prior into a posterior distribution. section 2.4.1
- Choose the next candidate by balancing exploration of uncertain regions with exploitation of settings that have performed well. section 2.4.1
Tuning a model
For machine-learning hyperparameter tuning, candidate settings might vary a random forest’s number of trees, a neural network’s learning rate, or a decision tree’s depth. Each evaluation may require training and testing the model. section 2.4.1





