Hyperparameter tuning selects model configuration values that optimize a performance measure such as predictive accuracy. It becomes difficult when each trial requires expensive training and only a few evaluations are available. section 1.1.1 section 1.1.2
A tuning task typically involves:
- Choosing configuration values such as neural-network depth, decision-tree depth, node size, split criterion, SVM regularization, or kernel parameters. section 1.1.1
- Querying the objective by training or evaluating the model at a selected setting and recording its performance. section 1.1.2
- Selecting promising next trials rather than exhaustively testing every setting when computation or time is limited. section 1.1.1 section 1.2.4
Choosing neural-network depth
A model reaches 72% accuracy with 5 layers, 81% with 10 layers, and 75% with 20 layers. Because training takes several days, the next trial should focus on promising values rather than scanning every possible depth. Bayesian optimization can balance exploring uncertain regions with exploiting regions that already look strong. section 1.1.1 section 1.2.4
Table 1.1 An example of a hyperparameter tuning task (view table figure)
Compare tested layer counts with their observed test accuracy before deciding where additional trials may be worthwhile. section 1.1.1
