5 Automating Hyperparameter Optimization

 

This chapter covers

  • Developing a process to manually optimize hyperparameters for deep learning networks.
  • Building automatic hyperparameter optimization with random search.
  • Formalizing automatic HPO by employing a grid search algorithm.
  • Applying evolutionary computation to HPO using particle swarm optimization.
  • Extending evolutionary HPO by using Evolutionary Strategies.
  • Applying Differential Evolution to HPO.

Over the last couple of chapters, we have been exploring various forms of evolutionary computation from genetic algorithms to particle swarm optimization and on to advanced methods like evolutionary strategies and differential evolution. All these EC methods we will continue to use through the rest of the book in some capacity to improve on deep learning. Combining into a process, we will colloquially call evolutionary deep learning.

However, before building a set of EDL solutions to various DL problems we would be remiss if we didn’t understand the problems we are trying to solve and how they are solved without EC. Afterall, EC tools are just one in a grand toolbox we can use to improve DL but they are not the only ones. Therefore, before we get into applying EC methods to HPO we are going to first look at the importance of hyperparameter optimization and some manual strategies. Second, when considering automated HPO we want to create a baseline by first reviewing other search methods such as random and grid search.

5.1 Option Selection and Hyperparameter Tuning

 
 

5.1.1 Tuning Hyperparameter Strategies

 
 

5.1.2 Selecting Model Options

 
 
 
 

5.2 Automating HPO with Random Search

 

5.2.1 Applying Random Search to HPO

 

5.3 Grid Search and HPO

 
 
 

5.3.1 Using Grid Search for Automatic HPO

 
 

5.4 Evolutionary Computation for HPO

 
 
 

5.4.1 Particle Swarm Optimization for HPO

 
 

5.4.2 Adding EC and DEAP to Automatic HPO

 
 

5.5 Genetic Algorithms and Evolutionary Strategies for HPO

 
 

5.5.1 Applying Evolutionary Strategies to HPO

 
 
 
 

5.5.2 Expanding Dimensions with Principal Component Analysis

 
 
 
 

5.6 Differential Evolution for HPO

 
 

5.6.1 Differential Search for Evolving HPO

 
 
 

5.7 Summary

 
 
 
sitemap

Unable to load book!

The book could not be loaded.

(try again in a couple of minutes)

manning.com homepage
test yourself with a liveTest