Precision at top N% is the proportion of positive cases among the highest-ranked N% of cases. In fraud detection, it measures how many flagged transactions are actual frauds. section 8.3.4
Evaluate the metric against the review scope:
- Rank transactions by predicted fraud probability, then select predefined top-percentage cutoffs. section 8.3.4
- Divide actual frauds in each selected group by the number of flagged transactions; compare with the overall fraud capture rate. section 8.3.4
- Choose a review scope that fits analyst capacity and the institution’s tolerance for false positives and missed fraud. section 8.3.4
Fraud review tradeoff
Reviewing the top 0.1% yields about 82.35% precision and captures 47.30% of fraud. Expanding to 0.2% lowers precision to 67.25% while raising capture to 77.70%; the suitable cutoff depends on review capacity and risk tolerance. section 8.3.4
Listing 8.6 Preparing data for top-N% analysis
# Create a DataFrame with true labels and predicted fraud probabilities for the test set
pred_df_eval_test = pd.DataFrame({'true_label': y_test, 'predicted_proba': y_pred_proba_test})
# Sort transactions by predicted fraud probability in descending order (highest probability first)
pred_df_sorted_eval_test = pred_df_eval_test.sort_values(by='predicted_proba', ascending=False) #1
# Define the percentage thresholds for our analysis
percentage_thresholds = [0.001, 0.002, 0.005, 0.01, 0.05] # Top 0.1%, 0.2%, 0.5%, 1%, 5%
total_test_samples = len(pred_df_sorted_eval_test)
total_actual_frauds_in_test = pred_df_sorted_eval_test['true_label'].sum() #2
print(f"Data prepared for Top-N% analysis. Total test samples: {total_test_samples}, Total actual frauds: {total_actual_frauds_in_test}")
See how test labels and predicted fraud probabilities are assembled, sorted, and prepared for top-percentage analysis. section 8.3.4
