Adversarial validation tests whether two datasets resemble each other by training a classifier to distinguish their samples. If the classifier struggles, their distributions are likely similar. section 2.3.3
A practical workflow for comparing original and synthetic tabular data:
- Combine both datasets and label each row by its source. section 2.3.3
- Encode categorical columns before fitting a binary classifier. section 2.3.3
- Use cross-validation and ROC AUC to measure how well the classifier separates the datasets. section 2.3.3
- Interpret an AUC near 0.5 as near-random discrimination; an AUC of 1 indicates perfect separation. section 2.3.3
Evaluating synthetic German Credit Risk data
A RandomForestClassifier produces an ROC AUC of approximately 0.567 when comparing synthetic and original records, indicating that the synthetic data is almost indistinguishable from the original. section 2.3.3
Listing 2.6 Testing a synthetic dataset using adversarial validation
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import roc_auc_score
X = pd.concat([synthetic_data, data]) #1
categorical_columns = [
"Sex", "Housing", "Saving accounts",
"Checking account", "Purpose"]
X_encoded = pd.get_dummies(X, columns=categorical_columns) #2
X= X.drop("index", axis="columns")
y = [0] * len(synthetic_data) + [1] * len(data)
model = RandomForestClassifier()
cv_preds = cross_val_predict(
model, #3
X_encoded, #3
y, #3
cv=5, #3
n_jobs=-1, #3
method="predict_proba" #3
) #3
roc_adv_score = roc_auc_score(y_true=y, y_score=cv_preds[:, 1]) #4
print(f"roc auc adv score: {roc_adv_score:0.3f}")
Inspect the runnable scikit-learn workflow for combining datasets, encoding categories, generating cross-validated predictions, and calculating ROC AUC. section 2.3.3
