Tree-structured Parzen estimator (TPE) is a Bayesian optimization sampler for sequential hyperparameter optimization. It proposes new hyperparameter values using previous trials and their evaluation results, helping search expensive black-box objectives with fewer evaluations than unguided search. section 5.2.2
A Hyperopt workflow assigns clear responsibilities:
- Wrap training and evaluation in an objective function that reads hyperparameter values. section 5.4.1
- Define a search space containing candidate choices and distributions for each hyperparameter. section 5.4.1
- Select TPE and configure the trial budget; Hyperopt passes sampled values to the objective function. section 5.4.1
Classifier selection
A search space can choose among naive Bayes, SVM, and decision-tree classifiers, while assigning model-specific parameters such as SVM kernel and width or tree criterion and depth. The objective returns model accuracy for comparison across trials. section 5.4.1
Listing 5.1 Getting started with Hyperopt
# Step 1: define an objective function
def objective(args):
model = train(args) #1
return evaluate(model) #1
# Step 2 define search space for hyperparameters
space = hp.choice('classifier_type', [ #2
{
'type': 'naive_bayes',
},
{
'type': 'svm',
'C': hp.lognormal('svm_C', 0, 1), #3
'kernel': hp.choice('svm_kernel', [ #3
{'ktype': 'linear'}, #3
{'ktype': 'RBF', #3
'width': hp.lognormal('svm_rbf_width', 0, 1)}, #3
]),
},
{
'type': 'dtree',
'criterion': hp.choice('dtree_criterion',
['gini', 'entropy']),
'max_depth': hp.choice('dtree_max_depth',
[None, hp.qlognormal('dtree_max_depth_int', 3, 1, 1)]),
'min_samples_split': hp.qlognormal(
'dtree_min_samples_split', 2, 1, 1),
},
])
#Step 3 start the hpo process execution
best = fmin(objective, space, algo=tpe.suggest,
max_evals=100) #4
Inspect the runnable Hyperopt pattern for defining the objective, nested classifier search space, and model-specific parameter distributions. section 5.4.1
Hyperopt supports serial execution and can coordinate parallel workers through a central database or Spark, making TPE usable beyond a single machine. section 5.4.1
