Bayesian optimization
Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
Bayesian Optimization in Action
A Bayesian optimization workflow is an iterative procedure that models an objective, selects candidate points, evaluates them, and updates the observed dataset.
- How it is used
- Targets expensive black-box objectives by choosing evaluations adaptively under a limited budget.
- How deep it goes
- Develops the GP–policy loop, acquisition scoring, exploration–exploitation tradeoffs, and Python scaffolding; later extends policies and specialized settings.
- Examples
- Uses a bounded one-dimensional objective, random and grid-search baselines, hyperparameter tuning, materials discovery, and a restaurant-choice analogy.
- What you can do after
- Build it from scratch as a GP-and-policy loop and apply it with GPyTorch and BoTorch to bounded black-box searches.
Read how Bayesian Optimization in Action explains Bayesian optimization →
Experimentation for Engineers
Bayesian optimization is a sequential optimization method that uses a probabilistic model and Bayesian updating to choose promising evaluations of an expensive objective.
- How it is used
- Reduces expensive business-metric measurements by automatically designing and analyzing experiments for system-parameter optimization.
- How deep it goes
- Explains initialization, surrogate modeling, acquisition optimization, exploration-exploitation balance, iteration, and convergence across seven parameters.
- Examples
- Uses a noisy JIT compiler simulator, center-point initialization at 0.5, boundary exploration at 0, and iterative CPU-time measurements.
- What you can do after
- Builds an automated loop that selects costly black-box experiments and stops when the modeled optimum stabilizes.
Read how Experimentation for Engineers explains Bayesian optimization →
Designing Deep Learning Systems: A guide for software engineers
Bayesian optimization is a sequential optimization technique that uses a probabilistic surrogate model to select promising evaluations of an expensive objective.
- How it is used
- Guides hyperparameter-trial selection as one of three HPO algorithm families.
- How deep it goes
- Introduces Bayesian optimization at a high level for building or setting up an HPO system, deferring mathematical reasoning.
- Examples
- Compares Bayesian and random-search data sampling across 10 trials.
- What you can do after
- Operate/configure it as an HPO algorithm choice under task and infrastructure constraints.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A Bayesian optimizer is an optimization method that uses outcomes from evaluated configurations to select promising future configurations.
- How it is used
- Tunes machine-learning hyperparameters when adaptive search can improve performance over a limited number of evaluations.
- How deep it goes
- Explains surrogate functions, acquisition, exploration, exploitation, and trial workflows without deriving Bayesian mathematics.
- Examples
- Demonstrates Optuna searches over XGBoost boosters, objectives, tree parameters, learning rates, sampling, and regularization on Tokyo Airbnb data.
- What you can do after
- Apply it via Optuna to define an objective, run adaptive trials, compare MAE, and inspect optimization history and parameter importance.
Machine Learning Algorithms in Depth
Bayesian optimization is a sequential optimization technique that uses a probabilistic surrogate model and an acquisition rule to optimize an expensive black-box function.
- How it is used
- Deploys active learning for hyperparameter optimization when objective evaluations are expensive.
- How deep it goes
- Explains Gaussian-process posteriors and EI/UCB selection, but does not provide acquisition formulas or derive Gaussian-process inference.
- Examples
- Uses cross-validated F1 optimization for SVM gamma and random forest estimator-count and depth parameters.
- What you can do after
- Apply it via bayes_opt with scikit-learn cross-validation to search SVM and random forest hyperparameters.
Read how Machine Learning Algorithms in Depth explains Bayesian optimization →
Related concepts
- black-box solverkind of · A black-box solver optimizes an objective through input-output evaluations without requiring access to its internal analytical form or dynamics.
- Bayes' theoremuses · Bayes' theorem is a probability rule that expresses a conditional probability in terms of a prior probability, a likelihood, and the evidence.
- MIPROv2used by · MIPROv2 is a DSPy optimizer that searches for effective instructions and demonstrations for language-model programs using performance feedback and a Bayesian proposal model.
- Gaussian processuses · A Gaussian process is a probability distribution over functions in which every finite collection of function values has a joint Gaussian distribution.
- surrogate modeluses · A surrogate model is an approximate model that estimates the behavior or outputs of a more complex, costly, or otherwise unavailable model, process, or function.
- optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.