A confusion matrix compares a classifier’s predictions with actual labels, showing which classes it identifies correctly and where it makes errors. section 8.3.3
For fraud classification, its four outcome categories are:
- True positive: fraud correctly identified. section 8.3.3
- True negative: legitimate transactions correctly identified. section 8.3.3
- False positive: a legitimate transaction incorrectly flagged as fraud. section 8.3.3
- False negative: fraud missed by the model. section 8.3.3
Fraud detection tradeoff
On the unseen test set, the tuned model identified 127 of 148 frauds (recall 0.86), while incorrectly flagging 456 legitimate transactions (precision 0.22). section 8.3.3
Listing 8.5 Comprehensive evaluation of the tuned model on the test set
# Make predictions on the (imbalanced) test set
y_pred_test = best_model_final.predict(X_test) #1
y_pred_proba_test = best_model_final.predict_proba(X_test)[:, 1] #2
print("\n--- Performance of Tuned LightGBM Model on UNSEEN TEST SET ---")
# Generate and plot the confusion matrix
print("\nConfusion Matrix (Test Set):")
cm_test = confusion_matrix(y_test, y_pred_test)
plt.figure(figsize=(6,4))
sns.heatmap(cm_test, annot=True, fmt='d', cmap='Oranges',
xticklabels=['Predicted Legit', 'Predicted Fraud'],
yticklabels=['Actual Legit', 'Actual Fraud'])
plt.ylabel('Actual Class', fontsize=12)
plt.xlabel('Predicted Class', fontsize=12)
plt.title('Confusion Matrix - Tuned Model (Test Set)', fontsize=14)
plt.show() #3
# Print a detailed classification report
print("\nClassification Report (Test Set):")
print(classification_report(y_test, y_pred_test, target_names=['Legitimate (0)', 'Fraudulent (1)'])) #4
# Calculate and print AUC scores
auc_roc_test = roc_auc_score(y_test, y_pred_proba_test)
precision_test, recall_test, _ = precision_recall_curve(y_test, y_pred_proba_test)
auc_pr_test = auc(recall_test, precision_test) #5
print(f"\nAUC-ROC Score (Test Set): {auc_roc_test:.4f}")
print(f"AUC-PR Score (Test Set): {auc_pr_test:.4f}")
print("--------------------------------------------------------------------")
Inspect the Python evaluation workflow that generates a confusion matrix for predictions on the imbalanced test set. section 8.3.3
