A model can score well and still disappoint in production. A performance metric is a defined measure used to judge whether the model is doing a useful job. Start with the project goal, then ask what observable result would show progress toward it. For a demand forecast, reducing prediction error is not the whole goal: the business also wants fewer gaps between delivered and sold items without running out of stock. section 5.2
Keep evaluation metrics distinct from training loss. A loss function guides model training and usually must be differentiable; a metric assesses performance and can be tailored to a business domain. The two may coincide, but choosing a familiar metric merely because it is available can steer development toward the wrong outcome. Agree on the measures and their purpose early enough to avoid costly redesign. section 5.0 section 5.2 section 5.3.3
Choose measures for each decision they need to support:
- Before launch, use offline metrics to compare models and estimate likely effects. Treat them as proxies when the result that matters can only be measured online. section 5.4 section 12.0
- In an A/B test, make key metrics decide success, control metrics check whether the experiment remains credible, and auxiliary metrics help explain what happened. Prefer online measures that are trustworthy, sensitive, interpretable, and quick to provide feedback. section 12.2.3
- After deployment, track prediction quality when ground truth arrives, including performance by segment. When feedback is delayed, monitor proxies such as changes in inputs or predictions; critical models may need more specific measures. section 14.4.3
Equal forecast error, different revenue
Two airline ticket forecasts can have the same weighted average percentage error—15%—yet make different pricing decisions. In a four-day illustration, one forecast tends to underpredict sales and yields $75,300; the other tends to overpredict and yields $83,900. A shared error score does not settle which forecast better serves the revenue goal. The revenue calculation is a simplified illustration, not a guarantee of production results. section 12.1.2
Metric context matters, too. For classification, accuracy, precision, and F1-score are possible quality measures; for regression, mean absolute error and root mean squared error are examples. In fraud detection, compare recall and precision alongside the false-positive rate rather than relying on one number. Ask how many genuine transactions are flagged, not only how many fraudulent ones are caught. section 14.4.3 section 5.2
Report results against the original goal, not just the model score. Offline testing can inform the decision to run an online experiment, but real-world feedback and A/B tests are needed to assess effects on business metrics. Keep the metric hierarchy available from design through reporting and monitoring. section 5.4 section 12.0
