5 Hyper parameter optimization (HPO) service
This chapter covers
- What are hyper parameters and why are they important?
- Two common approaches to HPO (hyper parameter optimization)
- Designing an HPO service
- Setting up your own HPO service with Kubeflow/Katib
- Three popular HPO libraries: Hyperopt, Optuna and Ray Tune
In the previous two chapters, we saw how models get trained: a training service manages training processes in a remote compute cluster with given model algorithms. But model algorithms and training services aren't all there is to model training. There's one more component we haven't discussed yet - hyper parameter optimization (HPO). Often overlooked is the fact that hyperparameter choices can influence model training results significantly, especially when these decisions can be automated using engineering methods.
Hyper parameters are parameters whose value must be set before the model training process starts. Learning rate, batch size, and number of hidden layers are all examples of hyper parameters. Unlike the value of model parameters –weights and bias, for example– hyper parameters cannot be learned during the training process.