Hyperparameter search space

As explained in Designing Deep Learning Systems: A guide for software engineers

The set of allowable values or value ranges from which hyperparameters may be selected during optimization.

A hyperparameter search space defines the values an optimization process may try while training a model. Rather than manually tweaking empirical settings, define candidate values or ranges, run trials, record evaluation results, and select the configuration that produces the best objective value. For learning rate and batch size, two six-value sets create 36 candidate pairs. section 5.2.1

Search spaces can combine discrete choices, continuous ranges, categorical options, and conditional values. Their size and structure directly affect cost: exhaustive evaluation becomes impractical as the number of hyperparameters or candidate values grows, while deep learning configurations can be high-dimensional and conditional. section 5.2.1 section 5.2.2

Choose a representation that fits the search algorithm and the training decision:

• Use finite sets for grid search. The algorithm forms the Cartesian product of each hyperparameter’s allowed values and evaluates the resulting configurations. section 5.2.2
• Use numeric intervals with a trial budget for random search. Each trial samples a value from each range until the budget is exhausted. section 5.2.2
• Use conditional branches when one choice enables its own parameters, such as classifier-specific SVM, decision-tree, or kernel settings. section 5.4.1
• Keep the search space separate from training code so an HPO service can support arbitrary frameworks, languages, and interchangeable search algorithms. section 5.3.1 section 5.3.2

Conditional classifier space

A classifier choice can include naive Bayes, SVM, and decision tree. Selecting SVM exposes parameters such as C, kernel type, and an RBF width drawn from a lognormal distribution; selecting a decision tree exposes criterion and maximum depth instead. This prevents trials from assigning irrelevant parameters to a chosen classifier. section 5.4.1

In a service workflow, submit training code, the hyperparameters and their value search spaces, an objective, and an HPO algorithm. A suggestion component selects a configuration; a job manager launches the trial and collects metrics, accuracy, progress, and tried values. This separation lets the service add algorithms without changing the user-facing experiment structure. section 5.3.2 section 5.3.1

Defining spaces in Python HPO tools

Optuna defines the space inside an objective function with categorical, floating-point, and integer suggestions, then trains and evaluates the sampled model. Ray Tune instead passes a search-space configuration to the training function; a learning rate may be sampled from a function and momentum uniformly from 0.1 to 0.9. Both patterns connect sampled values to an objective score. section 5.4.2 section 5.4.3

Figure 5.9 A general system design of an HPO service

This architecture locates the submitted search space between the API interface, algorithm adapter, and trial-execution manager. section 5.3.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains discrete, continuous, categorical, and conditional spaces through grid, random, and tool-defined configurations without mathematical derivations.

Examples

Uses learning rate and batch size grids, conditional classifier settings, Optuna suggestions, and Ray Tune learning-rate and momentum configurations.

What it assumes

Assumes a trainable model, evaluation objective, and selected hyperparameters; introduces search-space definition through concrete configurations.

What you can do after

Apply it via Hyperopt, Optuna, or Ray Tune by defining candidate values, ranges, and conditional parameters for model trials.

How it connects

Connects hyperparameter optimization with grid search, random search, Bayesian optimization, objective functions, trial execution, and distributed training.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA hyperparameter search space is the set of candidate values and constraints from which model configurations are selected during optimization.

Compare how 2 books explain hyperparameter search space →

Related concepts

• hyperparameter tuningused by · Hyperparameter tuning is the process of selecting or adjusting configurable model or procedure settings and evaluating them against a chosen objective.
• hyperparameter gridpart of · A hyperparameter grid is a structured set of candidate hyperparameter values whose combinations define configurations for systematic evaluation.
• Optunaused by · Optuna is an open-source Python framework for automated hyperparameter optimization through repeated trials that explore parameter configurations.
• grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
• search spacekind of · A search space is the set of possible states, choices, configurations, or candidate solutions that may be considered when exploring or solving a problem.
• hyperparameterhas part · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.