A fraud team cannot investigate every transaction. Recall, also called sensitivity, measures how much actual fraud its detector catches: actual frauds flagged ÷ all actual frauds. Higher recall means fewer missed frauds and less direct financial loss. Unlike accuracy, it remains informative when almost every transaction is legitimate; approving everything can produce 99% accuracy while catching no fraud. section 8.1.1
For a capacity-limited review queue, Fraud Capture Rate@TopN% is recall within the top N% of transactions ranked by predicted fraud probability. Sort labeled test transactions by score, select the highest-risk N%, count the actual frauds selected, and divide by all actual frauds in the test set. Compare several review sizes rather than assuming one cutoff is best. section 8.3.4
Choosing a review queue
Reviewing the top 0.1% flags 85 transactions and finds 70 frauds: 82.35% precision, but 47.30% recall. Expanding to the top 0.2% flags 171 and finds 115 frauds, raising recall to 77.70% while lowering precision to 67.25%. At the top 5%, recall reaches 89.86%, but precision falls to 3.11% across 4,272 alerts. More coverage therefore requires substantially more reviews, many of them false positives. section 8.3.4
Listing 8.6 Preparing data for top-N% analysis
# Create a DataFrame with true labels and predicted fraud probabilities for the test set
pred_df_eval_test = pd.DataFrame({'true_label': y_test, 'predicted_proba': y_pred_proba_test})
# Sort transactions by predicted fraud probability in descending order (highest probability first)
pred_df_sorted_eval_test = pred_df_eval_test.sort_values(by='predicted_proba', ascending=False) #1
# Define the percentage thresholds for our analysis
percentage_thresholds = [0.001, 0.002, 0.005, 0.01, 0.05] # Top 0.1%, 0.2%, 0.5%, 1%, 5%
total_test_samples = len(pred_df_sorted_eval_test)
total_actual_frauds_in_test = pred_df_sorted_eval_test['true_label'].sum() #2
print(f"Data prepared for Top-N% analysis. Total test samples: {total_test_samples}, Total actual frauds: {total_actual_frauds_in_test}")
The preparation code pairs test labels with predicted probabilities, sorts by risk, and records the total fraud count needed as recall’s denominator. section 8.3.4
Use recall alongside precision, not in isolation. The operating threshold must reflect investigation capacity, false-positive customer friction, missed-fraud losses, and risk appetite. F1 combines precision and recall when their error costs are roughly equal; AUC-PR assesses performance across decision thresholds on imbalanced data. section 8.1.1 section 8.5
