This chapter covers
- Developing a process to manually optimize hyperparameters for DL networks
- Building automatic hyperparameter optimization with random search
- Formalizing automatic HPO by employing a grid search algorithm
- Applying evolutionary computation to HPO using PSO
- Extending evolutionary HPO by using evolutionary strategies
- Applying DE to HPO
In the past few chapters, we have been exploring various forms of evolutionary computation, from genetic algorithms to particle swarm optimization and even advanced methods like evolutionary strategies and differential evolution. We continue to use all these EC methods through the rest of the book in some capacity to improve on DL. We combine these methods into a process we colloquially call evolutionary deep learning (EDL).
However, before building a set of EDL solutions to various DL problems, we would be remiss if we didn’t understand the problems we are trying to solve and how they are solved without EC. After all, EC tools are just a few in a grand toolbox we can use to improve DL. Therefore, before we get into applying EC methods to HPO, we first look at the importance of hyperparameter optimization and some manual strategies. Second, when considering automated HPO we want to create a baseline by first reviewing other search methods, such as random and grid search.