A hyperparameter is a value supplied when a model is created or trained, rather than a value learned directly from training data. It can configure execution, initialize behavior, or shape how an algorithm learns. Learned model parameters are the coefficients or weights determined through data and computation. section 4.1.2
Some hyperparameters support repeatable and practical runs: random_state fixes a random seed, n_jobs controls CPU use, and tolerance or caching can affect computation. Others influence model behavior, such as LogisticRegression C, which constrains pattern learning. section 4.1.2
Tune hyperparameters systematically instead of changing settings until results look acceptable:
- Define a search space: select settings to explore and meaningful value boundaries. section 6.3
- Use cross-validation and one or more evaluation metrics to compare candidate settings without overfitting the available data. section 6.3
- Match the search method to the problem: grid search exhaustively tests combinations and works best for few, discrete settings on manageable datasets. section 6.3.1
- Balance predictive gains against cost; adding trees may reduce random-forest variance, but excessive trees can lengthen training and inference without improvement. section 5.1.2
Grid-searching a pipeline
A GridSearchCV search can tune a logistic-regression pipeline with model__penalty, model__C, and model__l1_ratio. The model__ prefix addresses a parameter inside the pipeline’s model step; cross-validated results choose the strongest tested configuration. section 6.3.1
Hyperparameters remain central across model families. Gradient boosting uses settings such as learning_rate, n_estimators, tree depth, and leaf size; lower learning rates generally call for more trees. TabNet exposes optimizer, learning rate, scheduler, mask type, epochs, patience, and batch size. Keeping deep-learning values in a config file supports repeatable experiments without editing training code. section 6.3.5 section 8.3.1 section 9.4
Listing 6.12 Grid search
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import KFold, GridSearchCV
from sklearn.metrics import make_scorer
accuracy = make_scorer(accuracy_score)
cv = KFold(5, shuffle=True, random_state=0)
model = LogisticRegression(solver="saga", max_iter=5_000)
model_pipeline = Pipeline(
[('processing', lm_column_transform),
('model', model)])
search_grid = [
{"model__penalty": [None]},
{"model__penalty": ["l1", "l2"], "model__C": np.logspace(-4, 4, 10)},
{"model__penalty": ["elasticnet"], "model__C": np.logspace(-4, 4, 10),
"model__l1_ratio": [.1, .3, .5, .7, .9, .95, .99]},
] #1
search_func = GridSearchCV(estimator=model_pipeline, #2
param_grid=search_grid,
scoring=accuracy,
n_jobs=-1,
cv=cv)
search_func.fit(X=data, y=target_median)
print (search_func.best_params_) #3
print (search_func.best_score_) #4
Inspect a concrete search grid that compares regularization choices and ranges for C and l1_ratio inside a preprocessing-and-model pipeline. section 6.3.1
