An importance score summarizes how much input features influence a model’s predictions. In financial modeling, it offers a global view of model behavior for risk management and regulatory review; it does not explain an individual prediction. section 6.3.2 section 10.5.2
The score depends on its calculation method, so interpret the result in light of each method’s tradeoffs:
- Split-based importance counts how often a feature is used to split data. It is easy to read but can favor continuous or high-cardinality features even when their predictive power is low. section 6.3.2
- Gain-based importance measures the improvement in training loss from a feature’s splits. It is generally more reliable than split counts, but may favor features that capture localized noise. section 6.3.2
- Permutation importance measures how much model error increases after a feature’s values are shuffled. It is model-agnostic and intuitive, but can be costly on large datasets. section 6.3.2
- Mean absolute SHAP importance averages local SHAP values across customers. It accounts for feature interactions and is presented as a consistent option for regulatory reporting. section 6.3.2
In credit scoring, XGBoost importances from WoE-coded inputs indicate which bins or categories drive separation between defaults and nondefaults. They can be considered alongside reason codes for compliance; out-of-fold predictions, rather than optimistic in-sample averages, support validation. section 5.3.5 section 5.3.5
For market prediction, inspecting random-forest importances can help diagnose which features are driving changed predictions and inform intervention. section 10.5.2
Listing 5.14 The xgboost_model function
def xgboost_model(df_woe, target="target", folds=5, seed=2023):
import numpy as np
from sklearn.model_selection import StratifiedKFold
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score
xgb_models = [] #1
xgb_oof = []
predictions = np.zeros(len(df_woe), dtype=float)
f_imp = [] #2
X = df_woe.drop(columns=[target])
y = df_woe[target]
skf = StratifiedKFold(n_splits=folds, shuffle=True, random_state=seed)
for fold_idx, (train_idx, val_idx) in enumerate(skf.split(X, y)):
print(f"\n{'#'*24} Training FOLD {fold_idx+1} {'#'*24}")
X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]
model = XGBClassifier( #3
n_estimators=1000,
max_depth=4,
learning_rate=0.2,
colsample_bytree=0.67,
n_jobs=-1,
random_state=seed
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=0)
val_preds = model.predict_proba(X_val)[:, 1]
val_score = roc_auc_score(y_val, val_preds)
oof_array = np.vstack([val_idx, val_preds, y_val]).T
xgb_oof.append(oof_array)
feat_imp = dict(
zip(X_train.columns, model.feature_importances_)
)
f_imp.append(feat_imp)
print(f"{' '*16}Fold {fold_idx+1} AUC = {val_score:.5f}")
xgb_models.append(model)
if val_score > 0.917: #4
full_probs = model.predict_proba(X)[:, 1]
predictions += full_probs
predictions /= folds
fold_aucs = []
for arr in xgb_oof:
fold_auc = roc_auc_score(arr[:, 2], arr[:, 1])
fold_aucs.append(fold_auc)
mean_auc = np.mean(fold_aucs)
print("\n" + "*"*45)
print(f"Mean AUC: {mean_auc:.5f}")
return xgb_models, xgb_oof, predictions, f_imp
The cross-validation function collects feature importances for each fold, supporting inspection or averaging across trained models. section 5.3.5
