liveBook · Manning

Evaluation

As explained in Software Engineering for Data Scientists

Model evaluation is the assessment of a predictive model's performance on data, typically including data not used for development.

Model evaluation checks how well a trained predictive model performs on data that was not used for its development. Using fresh evidence helps determine whether the model can make reliable predictions beyond its training workflow. section 1.2.7

A practical evaluation workflow includes:

  • Reserve a fresh or hold-out dataset for evaluation after model development. section 1.2.7
  • Choose metrics suited to the prediction task, such as accuracy for correct classifications or correlation score for continuous predictions. section 1.2.7
  • Compare predictions with available ground truth and examine useful breakdowns, including precision and recall. section 1.4.4
  • Monitor performance after deployment with dashboards and notifications when results drop significantly. section 1.4.4

Customer churn predictions

For a churn model, compare each prediction with whether the customer actually churned. Precision measures how often predicted churn cases were correct, while recall measures how many customers who actually churned were successfully identified. section 1.4.4

Figure 1.4 A sample data science workflow. Data science workflows involve key steps like gathering data, exploring and cleaning the data, feature engineering, and model development. The model evaluation piece is also a highly important step, that we will cover in detail when we discuss model monitoring later in this book.
sample data science workflow

Use the workflow figure to place evaluation alongside data gathering, data exploration and cleaning, feature engineering, and model development. section 1.2.7

How Software Engineering for Data Scientists covers it

How deep it goes
Defines hold-out evaluation and several metrics, then extends the practice to ground-truth comparisons and production monitoring.
Examples
Uses hold-out datasets, continuous predictions, and customer churn cases to demonstrate metrics and ground-truth comparisons.
What it assumes
Assumes a trained model and evaluation data, while introducing hold-out testing, performance metrics, and ground-truth comparison.
What you can do after
Diagnose model performance with task-appropriate metrics, ground-truth comparisons, dashboards, and performance alerts.
How it connects
Connects model evaluation with model training, feature engineering, model monitoring, accuracy, precision, and recall.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.