Class imbalance occurs when outcome classes are represented unequally. The practical risk is that a classifier can favor the majority class, making accuracy less informative than metrics that assess ranking and minority detection. section 4.2.1
Choose evaluation measures according to the target distribution:
- Use accuracy for a balanced binary target, where positive and negative cases occur in nearly equal numbers. section 4.2.1
- Prefer ROC-AUC or Average Precision for an imbalanced target, such as one with substantially more cases below than above the mean. section 4.2.1
- Consider extremely randomized trees when random feature selection may prevent minority-class signals from being consistently displaced by majority-class signals. section 5.1.3
A skewed price target
Converting Airbnb listing prices into a binary target above or below the mean produces 34,016 negative cases and 14,879 positive cases. By contrast, splitting prices at the median produces 24,472 and 24,423 cases, making accuracy a safer measure for that target. section 4.2.1
Listing 5.5 ERTs classifier
from sklearn.ensemble import ExtraTreesClassifier
accuracy = make_scorer(accuracy_score)
cv = KFold(5, shuffle=True, random_state=0)
model = ExtraTreesClassifier(n_estimators=300,
min_samples_leaf=3,
random_state=0) #1
column_transform = ColumnTransformer(
[('categories', categorical_onehot_encoding, low_card_categorical),
('numeric', numeric_passthrough, continuous)],
remainder='drop',
verbose_feature_names_out=False,
sparse_threshold=0.0) #2
model_pipeline = Pipeline(
[('processing', column_transform),
('modeling', model)]) #3
cv_scores = cross_validate(estimator=model_pipeline,
X=data,
y=target_median,
scoring=accuracy,
cv=cv,
return_train_score=True,
return_estimator=True) #4
mean_cv = np.mean(cv_scores['test_score'])
std_cv = np.std(cv_scores['test_score'])
fit_time = np.mean(cv_scores['fit_time'])
score_time = np.mean(cv_scores['score_time'])
print(f"{mean_cv:0.3f} ({std_cv:0.3f})",
f"fit: {fit_time:0.2f}",
f"secs pred: {score_time:0.2f} secs") #5
Run an ExtraTreesClassifier pipeline on the balanced median-price target, then adapt the evaluation choice when the target distribution is skewed. section 5.1.3
