5 Automating Hyperparameter Optimization
This chapter covers
- Developing a process to manually optimize hyperparameters for deep learning networks.
- Building automatic hyperparameter optimization with random search.
- Formalizing automatic HPO by employing a grid search algorithm.
- Applying evolutionary computation to HPO using particle swarm optimization.
- Extending evolutionary HPO by using Evolutionary Strategies.
- Applying Differential Evolution to HPO.
Over the last couple of chapters, we have been exploring various forms of evolutionary computation from genetic algorithms to particle swarm optimization and on to advanced methods like evolutionary strategies and differential evolution. All these EC methods we will continue to use through the rest of the book in some capacity to improve on deep learning. Combining into a process, we will colloquially call evolutionary deep learning.
However, before building a set of EDL solutions to various DL problems we would be remiss if we didn’t understand the problems we are trying to solve and how they are solved without EC. Afterall, EC tools are just one in a grand toolbox we can use to improve DL but they are not the only ones. Therefore, before we get into applying EC methods to HPO we are going to first look at the importance of hyperparameter optimization and some manual strategies. Second, when considering automated HPO we want to create a baseline by first reviewing other search methods such as random and grid search.