Grid search evaluates hyperparameter configurations formed from the Cartesian product of user-specified candidate values, then runs an HPO trial for each configuration. section 5.2.2
Learning rate and batch size
Define learning-rate candidates {0.1, 0.005, 0.001} and batch-size candidates {10, 40, 100}. The grid includes combinations such as (0.1, 10), (0.1, 40), and (0.1, 100), which become trial configurations. section 5.2.2
Account for these operational trade-offs:
- Use bounded candidate sets when the search space is small and explicitly enumerable. section 5.2.2
- Expect evaluations to grow exponentially as hyperparameter count or search-space size increases. section 5.2.2
- Avoid assuming equal allocation is efficient: grid search can waste compute in nonoptimal regions while undersampling promising ones. section 5.2.2
Figure 5.4 Comparison of grid search and random search for minimizing a function with one important and one unimportant parameter. (Source: Figure 1.1. of “Hyperparameter Optimization,” by Matthias Feurer and Frank Hutter, in AutoML: Methods, Systems, Challenges, eds. Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren; Springer, 2019. www.automl.org/wp-content/uploads/2019/05/AutoML_Book_Chapter1.pdf)
Compare grid search with random search across important and unimportant parameters. section 5.2.2
