liveBook · Manning

Hyperparameter tuning

As explained in Software Engineering for Data Scientists

Hyperparameter tuning is the process of selecting configuration values for a machine learning algorithm to optimize model performance.

Hyperparameter tuning searches configuration values for a model and keeps the combination that performs best under a chosen evaluation metric. For a random forest or gradient boosting model, the search can vary settings such as tree depth, number of estimators, split requirements, feature count, learning rate, or subsampling. The goal is an optimized trained model rather than relying on one hand-picked configuration. section 4.2.2 section 5.2.1

Start with a parameter grid: a dictionary listing values to try for each setting. Configure the estimator, the grid, a scoring metric, a number of candidate combinations, a random seed, and `n_jobs` for parallel work. `RandomizedSearchCV` randomly selects combinations from the grid, trains them, and selects the optimized parameters. Randomized search can be faster than standard grid search when datasets and grids are large. section 4.2.2 section 5.2.1

A small model wrapper makes tuning repeatable across projects and model types:

  • Store the model, parameter grid, `n_jobs`, scoring metric, iteration count, and random state when creating the object. section 4.2.2
  • Expose a `tune(X_features, y)` method that fits `RandomizedSearchCV` and retains the trained search object. section 4.2.2
  • Expose prediction through the trained object, then inspect `best_estimator_` to see the selected hyperparameters. section 4.2.2
  • Reuse the same wrapper with random forest classifiers, gradient boosting classifiers, or random forest regressors by supplying a different estimator and suitable parameter grid. section 4.2.2

Customer churn search

For customer churn, define candidate values for `max_depth`, `min_samples_leaf`, `min_samples_split`, `max_features`, and `n_estimators`. Create a random forest tuning object with `scoring="roc_auc"`, `n_iter=10`, `random_state=0`, and four parallel jobs. Tune it on training features and labels, examine the best estimator, then generate predictions for both training and test features. The same pattern can be generalized to gradient boosting by changing the estimator and its grid. section 4.2.2

Tuning may take substantial time because it trains many candidate models. Set `verbose=1` to print search progress and completion time. Use `n_jobs` to run work across available CPU cores, but avoid choosing more cores than the machine has. On an artists-data hyperparameter search, the reported estimated runtime fell from 678.4 seconds with one core to 291.4 seconds with three, while actual times depend on the machine and other running applications. section 5.2.1 section 6.2.3

Table 6.2 This table summarizes the runtimes for performing the hyperparameter search using varying number of cores.
Number of cores Runtime (seconds)

1

678.4

2

561.3

3

291.4

Compare estimated hyperparameter-search runtimes as `n_jobs` changes from one to three CPU cores. section 6.2.3

How Software Engineering for Data Scientists covers it

How deep it goes
Works through parameter grids, randomized search, scoring, fitted estimators, predictions, monitoring, and parallel runtime; it does not explain search internals.
Examples
Uses customer churn classification and artists popularity regression with random forest and gradient boosting parameter grids.
What it assumes
Assumes Python, scikit-learn estimators, feature-label training data, and an evaluation metric while introducing a reusable tuning wrapper.
What you can do after
Apply it via scikit-learn RandomizedSearchCV to tune models, inspect the best estimator, monitor progress, and configure parallel jobs.
How it connects
Connects hyperparameter tuning to RandomizedSearchCV, grid search, random forests, gradient boosting, ROC-AUC, parallelization, and progress monitoring.

Also explained in

Compare how 22 books explain hyperparameter tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.