Balanced accuracy averages recall across classes, giving each class equal influence instead of allowing the largest class to dominate the score. It is especially useful when classes are imbalanced or when evaluating multiclass classifiers. section 8.3.3 section 9.3.1
Use balanced accuracy when:
- The class distribution is uneven and plain accuracy gives an overly optimistic impression of performance. section 8.3.3
- A multiclass model should perform consistently across classes rather than mainly on the most frequent class. section 4.4.2
- A model evaluation or feature-importance analysis must reflect the cost of missing examples from underrepresented classes. section 9.3.1
Comparing models on imbalanced data
Suppose positive examples occur at a 1:10 ratio with negative examples. Plain accuracy can hide weak performance on positives, while balanced accuracy accounts for the skew. In an ensemble comparison, test accuracy and balanced accuracy can therefore rank preprocessing and model combinations differently. For permutation feature importance, scoring='balanced_accuracy' makes a feature’s usefulness reflect this class-balanced evaluation goal. section 8.3.3 section 9.3.1
Listing 9.6 Computing permutation feature importance
from sklearn.inspection import permutation_importance
pfi = permutation_importance(
xgb, Xtst, ytst, #1
scoring='balanced_accuracy', #2
n_repeats=30) #3
Compute permutation feature importance with balanced accuracy as the scoring metric and repeat the shuffling procedure 30 times. section 9.3.1
