Model evaluation measures whether a trained model is useful for its intended goal. In a training pipeline, evaluation follows training and precedes testing, reporting, and artifact packaging. It uses metrics on data separate from fitting data, then connects prediction quality to business outcomes. section 10.1.1 section 12.1.1
A practical evaluation workflow includes:
- Define the system goal and formulate a hypothesis about how a change should affect the relevant metrics. section 12.1
- Reserve validation data for model comparison and hyperparameter tuning, and protect the test set for final evaluation only. section 7.2.1
- Measure offline metrics such as root mean square error, mean absolute percentage error, or normalized discounted cumulative gain. section 12.1.1
- Treat offline results as an approximation of online impact, because prediction metrics may have a weak or indirect relationship with business metrics. section 12.0 section 12.1.1
- Use online testing, commonly A/B testing, to estimate causal effects and report overall impact, segment effects, problems, and next steps. section 12.5
Choosing whether to deploy a new recommender
A team first compares the new recommender with the existing system using offline metrics. If the results improve, it checks historical relationships between offline and online metrics, then runs an A/B test to measure effects such as click-through rate or average revenue per user. Feedback loops and unknown alternatives shown to users make online results harder to interpret than offline scores. section 12.0 section 12.1.1 section 12.5
Figure 7.2 Standard by-the-book data split
Use the standard training/validation/test split to see how tuning data is separated from data reserved for final evaluation. section 7.2.1
