Random search

As explained in Designing Deep Learning Systems: A guide for software engineers

A hyperparameter search method that samples configurations randomly from a defined search space until a search budget is exhausted.

Random search is a model-free hyperparameter optimization method that samples configurations from a defined search space until the allocated search budget is exhausted. It makes no assumptions about training code and ignores correlations between trials. section 5.2.2

A practical random-search workflow consists of:

• Define ranges or distributions for each hyperparameter, such as learning rate [0.001, 0.1] and batch size [10, 100]. section 5.2.2
• Set a search budget, such as 100 trials, and independently sample a configuration for each trial. section 5.2.2
• Train and evaluate the model for every sampled configuration, then use the resulting trial outcomes to identify a strong configuration. section 5.2.2 section 5.4.1

Unlike grid search, random search does not enumerate a Cartesian product. This avoids grid search’s exponential evaluation growth as dimensions expand and can devote more trials to promising regions rather than distributing trials uniformly across all combinations. section 5.2.2

The method is straightforward to integrate into an HPO service because trials are independent. Hyperopt provides random search alongside TPE and adaptive TPE, and supports serial or parallel execution through workers coordinated by a central database or through Spark. section 5.4.1

Budgeted classifier search

A search can sample classifier_type from naive_bayes, svm, and dtree, while giving each classifier its own parameters. The objective function reads the sampled arguments, trains the model, and returns its evaluation result; the optimizer then repeats this process within the trial limit. section 5.4.1

Figure 5.4 Comparison of grid search and random search for minimizing a function with one important and one unimportant parameter. (Source: Figure 1.1. of “Hyperparameter Optimization,” by Matthias Feurer and Frank Hutter, in AutoML: Methods, Systems, Challenges, eds. Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren; Springer, 2019. www.automl.org/wp-content/uploads/2019/05/AutoML_Book_Chapter1.pdf)

Compare how random sampling behaves against grid search when one parameter matters more than another. section 5.2.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains sampling, search spaces, budgets, and trial execution while deferring mathematical reasoning.

Examples

Uses learning-rate and batch-size ranges, a 100-trial budget, and a conditional classifier search.

What it assumes

Assumes model training and evaluation code, while teaching search spaces, objective wrappers, and trial budgets at a high level.

What you can do after

Apply random search by defining an objective, specifying a search space, and running a bounded set of HPO trials.

How it connects

Contrasts random search with grid search and places it alongside Bayesian optimization, TPE, and model-free optimization.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Bayesian Optimization in ActionAn optimization strategy that evaluates randomly selected candidate locations without adaptively modeling the objective.

Compare how 2 books explain random search →

Related concepts

• grid searchalternative to · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
• Hyperbandalternative to · Hyperband is a hyperparameter-optimization algorithm that adaptively allocates computational resources among configurations while successively eliminating poorly performing candidates.
• lower confidence bounduses · A lower confidence bound is an acquisition function that combines a predicted response with a multiple of predictive uncertainty to favor promising or informative evaluations.
• training datasetuses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
• Bayesian optimizationcontrasts with · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• Hyperoptimplemented by · Hyperopt is an open-source Python library for automated hyperparameter optimization over configurable parameter spaces and search algorithms.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.