When one class dominates a classification dataset, ordinary cross-entropy can encourage mode collapse: predicting the popular class for nearly every input. Focal loss responds by making easy, high-confidence examples contribute less, leaving training capacity for difficult cases. section 5.1.1
Focal loss is a dynamically scaled form of cross-entropy. Its scaling factor approaches zero as confidence in the correct class rises, so well-classified examples are down-weighted while misclassified examples receive relatively more attention. section 5.1.1 section 5.3.2
Use focal loss when deciding how to make training respond to imbalanced data:
- Prefer it to ordinary binary cross-entropy when one class is much more common and difficult examples matter disproportionately. section 5.3.2
- Treat it as an alternative to manual undersampling, oversampling, or class weighting when those approaches require substantial tuning or prove unreliable. section 5.1.1
- Remember that focal loss addresses the classification component; other objectives, such as diversity, may require additional loss terms and tuned weights. section 5.3.2
PhotoStock search
A search engine learns from clicks and purchases, where positive outcomes may be rare. Binary cross-entropy is a baseline, but focal loss can emphasize the harder click or purchase examples. A separate diversity term may be added when reducing similarity among returned results is also important. section 5.3.2
Figure 5.3 The suggested focal loss function focuses more on misclassified examples while reducing the relative loss for well-classified examples (source: Lin et al.).
The figure shows focal loss assigning relatively greater importance to misclassified examples and reducing the relative loss for well-classified examples. section 5.1.1
