Surrogate model

As explained in Bayesian Optimization in Action

A surrogate model is a predictive model that approximates an objective function from observed evaluations.

A surrogate model is a predictive model fitted to observed evaluations of an objective function. It stands in for an expensive or otherwise difficult-to-query function: instead of knowing the function everywhere, optimization uses the model’s current belief to decide which evaluation should happen next. Because those predictions drive the decisions of Bayesian optimization, surrogate quality directly affects optimization performance. section 1.2.1

A useful intuition is to treat the surrogate as a map drawn from scattered measurements. A point-estimate regressor can predict one value at each location, but it does not express how uncertain that prediction is. Gaussian processes are commonly used as surrogates because they produce probability distributions rather than only single-valued predictions. This uncertainty quantification supports decisions that balance potential reward against risk. section 1.2.1

Figure 1.3 Non-Bayesian models, such as ridge regressors, make pointwise estimates, while GPs produce probability distributions as predictions. GPs thus offer a calibrated quantification of uncertainty, which is an important factor when making high-risk decisions.

Compare pointwise ridge-regression predictions with Gaussian-process predictive distributions to see why uncertainty is valuable when selecting evaluations. section 1.2.1

Surrogate can also describe a different object in multifidelity optimization: a low-fidelity function that approximates the true objective and is cheaper to evaluate. Unlike a predictive surrogate fitted from observations, this approximation is itself an available data source. It is inexact, but its shape or correlation with the objective can still make its evaluations informative. section 9.1 section 9.2.1

Expensive evaluations and cheap approximations

When tuning a neural-network architecture, training for five epochs can serve as a low-fidelity approximation to performance after fifty epochs. Likewise, a computer simulation can approximate a physical experiment. The optimization problem must decide when cheap but imperfect information is worthwhile and when to pay for an exact high-fidelity evaluation. section 9.1

In a multifidelity workflow, a predictive model can use observations from several fidelities by recording each evaluation’s relationship to the objective:

• Associate each function with a correlation value from 0 to 1; 1 denotes the objective itself, while higher values indicate more informative approximations. section 9.2.1
• Store the fidelity value as an extra feature column alongside each input, so the training data identifies which function produced each observation. section 9.2.1
• Train a Gaussian process whose kernel processes fidelity values, allowing predicted uncertainty to depend on an observation’s source and fidelity. section 9.7
• Choose queries by weighing information about the optimum against evaluation cost; low-fidelity queries can win when their lower cost offsets their inaccuracy. section 9.3.2

A fidelity score is not a generic measure of model accuracy. Here it encodes how closely a low-fidelity function relates to the ground truth and tells the Gaussian process how much to rely on that source. A zero-correlation approximation supplies no information about the objective, whereas the objective has correlation one with itself. section 9.2.1

Forrester-function dataset

For a one-dimensional objective and its approximation, observations can be sampled from both functions. A fidelity column marks values such as 0.5 for the approximation and 1.0 for the objective. The resulting dataset combines inexpensive and exact observations for multifidelity modeling. section 9.2.1

The choice is not always to query the cheap source. High-fidelity evaluations become preferable when only low-fidelity observations have been collected or when low-fidelity evaluations cost nearly as much as high-fidelity ones. Cost-aware optimization therefore treats a surrogate’s usefulness as information relative to the price of obtaining it. section 9.3.2

How Bayesian Optimization in Action covers it

How deep it goes

Explains Gaussian-process probabilistic predictions, then develops multifidelity data formatting, correlation, cost modeling, and information-per-cost selection.

Examples

Uses ridge regression versus a Gaussian process, a Forrester approximation, neural-network early training, simulations, and Branin approximations.

What it assumes

Introduces Gaussian processes through prediction intuition without assuming prior Gaussian-process knowledge.

What you can do after

Evaluate/understand when predictive uncertainty and cheaper approximate evaluations should guide expensive black-box optimization.

How it connects

Connects Gaussian processes, Bayesian optimization, uncertainty quantification, multifidelity optimization, Max-value Entropy Search, and cost models.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Experimentation for EngineersA surrogate function is a model that approximates a measured response as a function of parameter values and estimates the response at unmeasured values.
• 

  liveBookmanning.com
  Outlier Detection in PythonA proxy model is an approximate model that imitates the behavior of another model or detector for analysis or explanation.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA surrogate model is an interpretable approximation trained to reproduce the behavior of a more complex model in a specified region or representation.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA surrogate function is an approximate model of an objective function that predicts objective behavior across untested parameter settings.

Compare how 5 books explain surrogate model →

Related concepts

• Optunaused by · Optuna is an open-source Python framework for automated hyperparameter optimization through repeated trials that explore parameter configurations.
• Bayesian optimizationused by · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• interpretabilityused by · Interpretability is the degree to which people can understand a model’s behavior, outputs, predictions, or the factors that produce them.
• black boxuses · A black box is a system or component whose internal workings are treated as opaque while its inputs, outputs, or observable behavior are considered.
• Gaussian processproduced by · A Gaussian process is a probability distribution over functions in which every finite collection of function values has a joint Gaussian distribution.
• labelcontrasts with · A label is a value or annotation associated with an item that identifies its target category, outcome, or reference answer for learning or evaluation.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.