Hyperparameter tuning, or hyperparameter optimization (HPO), searches for values that produce the best model under a predefined objective. Hyperparameters are set before training rather than learned during it; common examples include learning rate, batch size, epoch count, hidden-layer count, and regularization. Their choices can affect model quality as well as training time and memory use. section 5.0 section 5.2.1
An HPO experiment is a trial loop. Each trial trains the same model on the same dataset with a different configuration, evaluates the result, and records metrics. The loop stops when a budget is exhausted or a performance target is reached; the configuration behind the best evaluation becomes the selected set. An algorithm can generate each new configuration, replacing slow, intuition-driven manual selection. section 5.2.1 section 5.4
Figure 5.1 This high-level view of the HPO workflow shows that the process is essentially an experiment to find the optimal hyperparameter values.
Use the workflow diagram to trace how training, evaluation, stopping conditions, and new hyperparameter suggestions repeat across trials. section 5.2.1
Choose a search strategy according to search-space size, available trial budget, and execution constraints:
- Grid search enumerates the Cartesian product of specified values. It is straightforward but grows exponentially as parameters or value ranges increase. section 5.2.2
- Random search samples configurations until its budget is spent, rather than evaluating every grid combination. section 5.2.2
- Bayesian, model-free, and multifidelity methods are the three broad algorithm categories. No single algorithm fits every task or constraint. section 5.2.2 section 5.5
- Early-stopping schedulers can terminate unpromising trials and redirect time and resources toward more promising ones. section 5.4.3
A lightweight library approach embeds the objective function, search space, and optimization call in a training application. It is flexible and quick for local or small-cluster work, but scaling workers, repeating setup after code changes, and recovering from failures become operational burdens. A service approach instead accepts a job request and runs trials remotely as a black-box system, making it better suited to large teams and production use. section 5.2.3 section 5.5
Use the reference architecture to identify the API entry point, job manager, and algorithm adapter in an HPO service. section 5.3.2
A general HPO service separates submission, orchestration, and algorithm selection. Its API receives training code, hyperparameter search spaces, an objective, and an algorithm. The job manager launches and monitors trials, collects metrics and tried values, configures distributed training, and recovers failures. The suggestion maker wraps algorithms behind a uniform interface, allowing new algorithms to be registered without changing the user-facing workflow. section 5.3.2 section 5.3.1
Library-based tuning pattern
Define an objective function that trains and evaluates a model from sampled values; define each parameter's search space; then run a selected search algorithm for a trial limit. Hyperopt illustrates this pattern by selecting among naive Bayes, SVM, and decision-tree classifiers, while Optuna places sampling logic in an objective function and tracks the run as a study. section 5.4.1 section 5.4.2
For distributed experimentation, Ray Tune links an objective function and search space to Ray's execution layer. It can distribute work across nodes and GPUs, manage distributed computing and fault tolerance, and use schedulers such as ASHA to stop weak trials early. In a production training workflow, the workflow service can submit optimization work to an HPO service instead of directly invoking the training service. section 5.4.3 section 10.2.4
