Black-box solver

As explained in Bayesian Optimization in Action

Black-box optimization is optimization of an objective whose internal form is unavailable to the optimizer and can only be explored through evaluations.

A black-box solver searches for the input that gives the best output when the objective’s formula is unknown. It can only query the objective: choose a setting, run the process, and observe the resulting value. The goal is the global optimum, but each query may consume substantial time, compute, money, or experimental effort. section 1.1.2

Hyperparameter tuning makes the constraint tangible. Training models with 5, 10, or 20 layers reveals some accuracies, but not the accuracy at untried depths or whether a better depth lies outside the tested range. Exhaustively evaluating settings is often impractical, so the solver must spend a limited evaluation budget carefully. section 1.1.1 section 1.1.2

Figure 1.2 The framework of a black box optimization problem. We repeatedly query the function values at various locations to find the global optimum.

View the optimization loop: repeatedly query selected inputs, record their values, and use those observations to seek the global optimum. section 1.1.2

A costly black-box problem has three defining parts:

• An objective function whose value can be observed but whose underlying relationship between inputs and outputs is unavailable. section 1.1.2
• A search space of settings or parameters to try, such as neural-network depth, material composition, or airplane design variables. section 1.1.1 section 1.1.3 section 7.5
• An evaluation cost that rules out cost-blind exhaustive search and makes each choice of query consequential. section 1.1.2 section 1.1.3

Ten evaluations of an unknown function

For a one-dimensional objective on −5 to 5, only 10 evaluations are allowed. Random search may spend samples in unpromising regions, while grid search can still be inefficient at locating a good region. Bayesian optimization instead adapts later queries to earlier observations, balancing investigation of promising areas with exploration elsewhere until it finds the global maximum in the illustrated run. section 1.2.4 section 1.4

Bayesian optimization is a solver for expensive black-box problems. A Gaussian process predicts objective values and uncertainty; a policy uses those predictions to decide which point is most helpful to evaluate next. This adaptive loop is designed to find an optimum quickly and can outperform random or grid search when observations are scarce. section 1.4

How Bayesian Optimization in Action covers it

How deep it goes

Defines objectives, queries, costs, and global optima, then demonstrates adaptive search against random and grid baselines.

Examples

Uses neural-network layer tuning, materials and drug discovery, a one-dimensional objective, and simulated airplane design optimization.

What it assumes

Introduces black-box objectives through input-output queries before using Gaussian-process predictions and policies.

What you can do after

Evaluate/understand when costly evaluations make adaptive Bayesian optimization preferable to random or grid search.

How it connects

Connects objective functions, global optimization, hyperparameter tuning, random search, grid search, Gaussian processes, and Bayesian optimization.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• Bayesian optimizationincludes · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• constraint functionuses · A constraint function is a function whose value expresses whether decision variables satisfy a feasibility restriction in an optimization problem.
• objective functionuses · An objective function is a function or criterion that assigns values to candidate solutions or parameter settings and is optimized by minimizing or maximizing those values.
• hyperparameter tuningincludes · Hyperparameter tuning is the process of selecting or adjusting configurable model or procedure settings and evaluating them against a chosen objective.
• multifidelity optimizationincludes · Multifidelity optimization combines evaluations of an objective at different levels of accuracy and cost to reduce optimization expense.
• parallelismaddressed by · Parallelism is the concurrent execution or progress of multiple computational operations or tasks.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.