A classifier assigns an input to a category. The useful question is not just whether it predicts correctly, but whether its categories and mistakes match the job. A fraud classifier, for example, distinguishes fraudulent from genuine transactions. A medical-image classifier might answer whether an image shows a malignancy—or which kind it shows. Those are different labeling decisions, with different demands on the people preparing the data. section 5.2 section 6.2.4
Before choosing how to evaluate a classifier, pin down two decisions:
- Choose the labels the product needs. Binary labels can record whether a malignancy is present; multiclass labels can record its kind. More detailed alternatives, such as detection or segmentation, require different tools, expertise, and time. section 6.2.4
- Choose metrics for the operational goal. Fraud rates can change sharply from day to day, so precision alone may give a misleading impression. Compare recall and false-positive rate too, and ask how many legitimate transactions the classifier flags. section 5.2
Same recall, different workload
Suppose 100 of 100,100 transactions are fraudulent. Two classifiers each catch 80 frauds, giving both 80% recall. One also flags 20 genuine transactions; the other flags 920. The second catches no additional fraud but sends far more legitimate transactions for attention. Changing the number of frauds while holding genuine transactions steady also changes the apparent precision of these predictions. section 5.2
Label design sets a limit on what the classifier can deliver. If the needed detail is too costly or difficult to label reliably, consider whether partial or inaccurate labels and a weakly supervised approach are appropriate. Decide that from the product goal, not from the apparent simplicity of a classification label. section 6.2.4
