Bayesian optimization

As explained in Designing Deep Learning Systems: A guide for software engineers

Bayesian optimization is a sequential optimization technique that uses a probabilistic surrogate model to select promising evaluations of an expensive objective.

Bayesian optimization is a sequential optimization technique for expensive objectives. In hyperparameter optimization (HPO), it is one of three major algorithm families, alongside model-free and multifidelity optimization. Its role is to guide trial selection rather than treat every candidate configuration equally. section 5.2.2

Use it as one selectable strategy within an HPO system, not as a universally best default. Algorithm choice depends on the HPO task and its constraints; the practical objective is to find hyperparameters that minimize a predefined loss for a training workload. section 5.2.2 section 5.5

Figure 5.5 A data sampler comparison of random search (a) and Bayesian approach (b) using 10 trials

Compare how a Bayesian approach samples trials against random search when both are limited to 10 trials. section 5.2.2

For production HPO, pair algorithm selection with execution infrastructure: trials can be parallelized, distributed training can be introduced, and unpromising trials can be stopped early. Larger teams may use an HPO service for automated black-box operation and resource management. section 5.5

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Introduces Bayesian optimization at a high level for building or setting up an HPO system, deferring mathematical reasoning.

Examples

Compares Bayesian and random-search data sampling across 10 trials.

What it assumes

Teaches the algorithm family from high-level HPO engineering context without requiring mathematical reasoning.

What you can do after

Operate/configure it as an HPO algorithm choice under task and infrastructure constraints.

How it connects

Connects Bayesian optimization with hyperparameter optimization, model-free optimization, multifidelity optimization, random search, and HPO services.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Bayesian Optimization in ActionA Bayesian optimization workflow is an iterative procedure that models an objective, selects candidate points, evaluates them, and updates the observed dataset.
• 

  liveBookmanning.com
  Experimentation for EngineersBayesian optimization is a sequential optimization method that uses a probabilistic model and Bayesian updating to choose promising evaluations of an expensive objective.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA Bayesian optimizer is an optimization method that uses outcomes from evaluated configurations to select promising future configurations.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthBayesian optimization is a sequential optimization technique that uses a probabilistic surrogate model and an acquisition rule to optimize an expensive black-box function.

Compare how 5 books explain Bayesian optimization →

Related concepts

• black-box solverkind of · A black-box solver optimizes an objective through input-output evaluations without requiring access to its internal analytical form or dynamics.
• Bayes' theoremuses · Bayes' theorem is a probability rule that expresses a conditional probability in terms of a prior probability, a likelihood, and the evidence.
• MIPROv2used by · MIPROv2 is a DSPy optimizer that searches for effective instructions and demonstrations for language-model programs using performance feedback and a Bayesian proposal model.
• Gaussian processuses · A Gaussian process is a probability distribution over functions in which every finite collection of function values has a joint Gaussian distribution.
• surrogate modeluses · A surrogate model is an approximate model that estimates the behavior or outputs of a more complex, costly, or otherwise unavailable model, process, or function.
• optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.