Stability selection evaluates whether features remain selected across many resampled datasets. It is especially useful when L1 regularization gives different answers for highly correlated features, because chance can influence which similar variables receive non-zero coefficients. section 6.2.1
A practical workflow is:
- Choose a sparse selector such as L1-penalized logistic regression. section 6.2.1
- Fit it repeatedly on bootstrap samples across a grid of regularization values. section 6.2.1
- Keep features selected frequently enough to be considered robust, using the result as an initial reduction rather than a minimal feature set. section 6.2.1
Airbnb feature screening
A pipeline can standardize transformed Airbnb NYC data, fit 100 bootstrapped L1 logistic-regression models for each regularization setting, and inspect selection frequencies for categorical and numeric predictors. section 6.2.1
Listing 6.9 Stability selection
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import BaggingClassifier
lambda_grid=np.logspace(-4, -1, 10) #1
sparse_coef = list()
for modeling_c in lambda_grid:
estimator = LogisticRegression(
solver='liblinear',
penalty='l1',
C=modeling_c
) #2
model = BaggingClassifier(
estimator,
n_estimators=100,
bootstrap=True
) #3
model_pipeline = Pipeline(
[('processing', lm_column_transform),
('standardize', StandardScaler()), #4
('modeling', model)])
model_pipeline.fit(data, target_median)
sparse_coef += [estimator.coef_.ravel() for estimator in
model_pipeline["modeling"].estimators_]
epsilon = 1e-2 #5
threshold = 0.5 #6
non_zero = (np.abs(sparse_coef) > epsilon).mean(axis=0)
feature_names = model_pipeline["processing"].get_feature_names_out()
print(non_zero)
print(feature_names[non_zero > threshold])
Inspect the runnable pipeline that combines a regularization grid, L1 logistic regression, bootstrap ensembles, preprocessing, and standardization. section 6.2.1
