7 Customizing the search method of AutoML

 

This chapter covers

  • Understanding sequential search methods
  • Customizing a random search method
  • Vectorizing hyperparameters for the model-based search method
  • Understanding and implementing a Bayesian optimization search method
  • Understanding and implementing an evolutionary search method

In this chapter, we will explore how to customize a sequential search method to iteratively explore the hyperparameter search space and discover better hyperparameters. You will learn how to implement different sequential search methods for selecting pipelines from the search space in each trial. These search methods fall into the following two categories:

  • History-independent sequential search methods cannot be updated during the search process. For example, grid search, which we looked at in chapter 2, traverses all the possible combinations of values in the candidate hyperparameter sets, and in chapter 6, we used the random search method to select hyperparameter combinations from the search space randomly. These are the two most representative history-independent methods. Some other advanced random search methods make use of history, such as the quasi-random search method using Sobol sequences (http://mng.bz/6Z7A), but here we’ll consider only the vanilla uniform random search method.
  • History-dependent sequential search methods, such as Bayesian optimization, are able to improve the effectiveness of the search by leveraging the previous results.

7.1 Sequential search methods

7.2 Getting started with a random search method

7.3 Customizing a Bayesian optimization search method

7.3.1 Vectorizing the hyperparameters

7.3.2 Updating the surrogate function based on historical model evaluations

7.3.3 Designing the acquisition function

7.3.4 Sampling the new hyperparameters via the acquisition function

7.3.5 Tuning the GBDT model with the Bayesian optimization method

7.3.6 Resuming the search process and recovering the search method

7.4 Customizing an evolutionary search method