Bayesian optimization sequentially searches an expensive black-box objective by modeling its performance with a Gaussian process (GP), then selecting new parameter values through an acquisition rule. It balances exploration, which reduces uncertainty, with exploitation of promising results. section 7.4.1
A typical optimization loop proceeds as follows:
- Treat the performance function as a GP, f(x) ∼ GP(m(x), κ(x, x')). section 7.4.1
- Evaluate selected hyperparameter values and maintain the posterior distribution as observations accumulate. section 7.4.1
- Choose the next point by optimizing expected improvement (EI) or the Gaussian process upper confidence bound (UCB). section 7.4.1
- Repeat the process to find a strong result with relatively few evaluations, accepting additional computation for selecting each point. section 7.4.1
SVM and random forest search
A classification experiment uses Bayesian optimization to maximize cross-validated F1 score. For an SVM with an RBF kernel, gamma is optimized; for a random forest classifier, hyperparameters including the number of estimators and maximum depth are searched. section 7.4.1
Figure 7.12 Bayesian optimization algorithm
Use the algorithm diagram to inspect the sequential relationship between observations, the probabilistic model, acquisition, and the next evaluation. section 7.4.1
Listing 7.7 Bayesian optimization for SVM and RFC
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier as RFC
from sklearn.svm import SVC
from bayes_opt import BayesianOptimization
np.random.seed(42)
# Load data set and target values
data, target = make_classification(
n_samples=1000,
n_features=45,
n_informative=12,
n_redundant=7
)
target = target.ravel()
def svccv(gamma): #1
val = cross_val_score(
SVC(gamma=gamma, random_state=0),
data, target, scoring='f1', cv=2
).mean()
return val
def rfccv(n_estimators, max_depth): #2
val = cross_val_score(
RFC(n_estimators=int(n_estimators),
max_depth=int(max_depth),
random_state=0
),
data, target, scoring='f1', cv=2
).mean()
return val
if __name__ == "__main__":
gp_params = {"alpha": 1e-5}
#SVM
svcBO = BayesianOptimization(svccv,
{'gamma': (0.00001, 0.1)})
svcBO.maximize(init_points=3, n_iter=4, **gp_params)
#Random Forest
rfcBO = BayesianOptimization(
rfccv,
{'n_estimators': (10, 300),
'max_depth': (2, 10)
}
)
rfcBO.maximize(init_points=4, n_iter=4, **gp_params)
print('Final Results')
print(svcBO.max)
print(rfcBO.max)
Use the implementation listing to examine Bayesian hyperparameter search for SVM and random forest classifiers with scikit-learn cross-validation and the bayes_opt package. section 7.4.1
