11 Hyperparameter optimization

In chapter 10, you integrated your DC taxi model with the PyTorch Lightning framework, factoring out boilerplate engineering code and paving the way to hyperparameter optimization support. In this chapter, you are going to adopt Optuna, a hyperparameter optimization framework, to progress beyond a trial-and-error approach to selection of your machine learning hyperparameter values. You will train a collection of DC taxi model instances based on the hyperparameter values selected using Optuna’s Tree-Structured Parzen Estimator(TPE) that fits a Gaussian mixture model (GMM) to the hyperparameters in your machine learning system. The performance of these model instances is compared using various Optuna visualization plots.

11.1 Hyperparameter optimization with Optuna

This section teaches about Optuna for hyperparameter optimization (HPO) and how to add support for HPO to the DC taxi fare estimation model.

11.1.1 Understanding loguniform hyperparameters

11.1.2 Using categorical and log-uniform hyperparameters

11.2 Neural network layers configuration as a hyperparameter

11.3 Experimenting with the batch normalization hyperparameter

11.3.1 Using Optuna study for hyperparameter optimization

11.3.2 Visualizing an HPO study in Optuna

Summary