ROC AUC, also called area under the curve (AUC), evaluates a binary classifier by measuring how well its predicted probabilities distinguish positive from negative instances. A score of 1 represents a perfect classifier, while 0.5 represents performance no better than random guessing. section 2.3.3
Use ROC AUC when classification performance depends on ordering and class balance makes accuracy less suitable. It is especially useful for an imbalanced binary target, where ranking predictions can be more informative than counting correct labels. section 4.2.1
Testing synthetic data
Adversarial validation combines synthetic and original records, labels their source, and trains a classifier to distinguish them. A ROC AUC of approximately 0.567 indicates that the synthetic data is almost indistinguishable from the original data. section 2.3.3
Listing 2.6 Testing a synthetic dataset using adversarial validation
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import roc_auc_score
X = pd.concat([synthetic_data, data]) #1
categorical_columns = [
"Sex", "Housing", "Saving accounts",
"Checking account", "Purpose"]
X_encoded = pd.get_dummies(X, columns=categorical_columns) #2
X= X.drop("index", axis="columns")
y = [0] * len(synthetic_data) + [1] * len(data)
model = RandomForestClassifier()
cv_preds = cross_val_predict(
model, #3
X_encoded, #3
y, #3
cv=5, #3
n_jobs=-1, #3
method="predict_proba" #3
) #3
roc_adv_score = roc_auc_score(y_true=y, y_score=cv_preds[:, 1]) #4
print(f"roc auc adv score: {roc_adv_score:0.3f}")
Inspect a scikit-learn workflow that combines datasets, encodes categorical columns, generates cross-validated predictions, and calculates ROC AUC. section 2.3.3
