Fraud is usually rare, and its share of transactions can change sharply from day to day. That is class imbalance: one classification class has far fewer examples than another. It matters both when training a model and when deciding whether its reported performance will hold up in production. section 5.2
Check imbalance at three points in the ML workflow:
- Training: with a heavily overrepresented class, a model trained using cross-entropy loss may learn to output the popular class for every input. Undersampling, oversampling, and class weights are options, but can require substantial tuning. Focal loss instead reduces the contribution of easy, confidently classified examples so training focuses more on hard ones. section 5.1.1
- Evaluation: ask whether a metric changes when class balance changes. Precision can shift even if a model’s rates of detecting fraud and falsely flagging legitimate transactions stay the same. Recall and specificity are insensitive to that change in balance. Choose metrics for the decision they support, not just for familiarity. section 5.2
- Data preparation: compare counts at each step that supplies labels. Search results are shown more often than they are clicked, and clicks occur more often than purchases; subsampling may be needed to adjust those balances. section 6.7.2
When purchases are too rare
Suppose 10,000 displayed offers produce about 300 clicks but only 9 purchases. Predicting purchases directly leaves few positive examples and makes A/B tests slow. Click-through rate offers a more frequent proxy for purchase conversion, provided the relationship is checked and the eventual purchase and sales outcomes remain in view. More plentiful clicks make iteration easier; they do not make a click equivalent to a purchase. section 5.2.2
For a search system optimizing clicks and purchases, binary cross-entropy is a starting point, while focal loss is a candidate if those labels are imbalanced. Treat it as a choice to test against the desired outcomes, not an automatic fix for every uneven dataset. section 5.3.2
Figure 5.3 The suggested focal loss function focuses more on misclassified examples while reducing the relative loss for well-classified examples (source: Lin et al.).
The focal-loss figure contrasts the relative loss assigned to misclassified and well-classified examples, clarifying what it changes during training. section 5.1.1
