Model evaluation checks how well a trained predictive model performs on data that was not used for its development. Using fresh evidence helps determine whether the model can make reliable predictions beyond its training workflow. section 1.2.7
A practical evaluation workflow includes:
- Reserve a fresh or hold-out dataset for evaluation after model development. section 1.2.7
- Choose metrics suited to the prediction task, such as accuracy for correct classifications or correlation score for continuous predictions. section 1.2.7
- Compare predictions with available ground truth and examine useful breakdowns, including precision and recall. section 1.4.4
- Monitor performance after deployment with dashboards and notifications when results drop significantly. section 1.4.4
Customer churn predictions
For a churn model, compare each prediction with whether the customer actually churned. Precision measures how often predicted churn cases were correct, while recall measures how many customers who actually churned were successfully identified. section 1.4.4
Figure 1.4 A sample data science workflow. Data science workflows involve key steps like gathering data, exploring and cleaning the data, feature engineering, and model development. The model evaluation piece is also a highly important step, that we will cover in detail when we discuss model monitoring later in this book.
Use the workflow figure to place evaluation alongside data gathering, data exploration and cleaning, feature engineering, and model development. section 1.2.7




























