chapter seven
7 Hyper parameter optimization service
This chapter covers
- What are hyper parameters and why are they important?
- Introduction to HPO (hyper parameter optimization) two common approaches
- Designing an HPO service
- Setting up your own HPO service with Kubeflow/Katib
- Introduce three popular HPO libraries: Hyperopt, Optuna and Ray Tune
Hyper parameters are parameters whose value must be set before the model training process starts. Learning rate, batch size, and number of hidden layers are all examples of hyper parameters. Unlike the value of model parameters –weights and bias, for example– hyper parameters cannot be learned during the training process.
Much research reveals that the chosen value of hyperparameters can impact both the quality of the model training as well as the time and memory requirements of the training algorithm. Therefore hyperparameters have to be tuned to be optimal for model training. Nowadays, HPO (hyper parameter optimization) has become a standard step in the deep learning model development process.