Evaluation asks whether a model is useful for its intended application, not merely whether it scores well on a test set. A classifier can be 99% accurate and still add no value if 99% of the test examples belong to one class. Compare performance with the business requirement and with simple baselines before investing in more complex models. section 6.3
Make evaluation a planned project activity:
- Define and implement baselines early. A quick model on a small data sample can establish a starting point, even if it overfits or misses some complexity. section 6.3
- Document the modelling process, set up a test environment, and develop tests for functional performance. section 7.1
- Screen out obviously poor candidates before committing to full evaluation. Use fresh, unseen data or, where necessary, a live part of the application for the candidates worth testing. section 7.1 section 8.1
Separate rapid model development from rigorous test and selection. A team may produce many candidate generations in the time required for one full evaluation. Unseen data may be limited, while live tests can affect people or customers. Releasing to a controlled test environment also takes time. Select candidates carefully rather than applying the most expensive tests to every iteration. section 8.1
A churn-model baseline
Before selecting a sophisticated customer-churn model, compare it with a handcrafted classifier based on contract expiry, estimated household income, or monthly expenditure. Ask how much improvement is needed to deliver the project's required return on investment—not just whether the new model wins by a small margin. section 6.3
Keep evidence that others can inspect. Evaluation supports both model selection and a defensible account of why a model meets stakeholder needs. Production documentation should also say how to evaluate model behavior and what matters about the value delivered to users. Without that record, a promising result is harder to verify, govern, or maintain. section 7.0 section 9.5.2
Table 7.2 Backlog for Sprint 2: modelling and evaluation (view table figure)
| Task # |
Item |
|---|---|
| S2.1 |
Create a feature engineering plan, then share and review it with team. Implement a feature-engineering pipeline. Design and add data augmentation. |
| S2.2 |
Create the model’s design and document it. Consider the forces acting on the model’s design. Decide on the decomposition of tasks to create your overall design. Choose component parts based on the required inductive biases given your data. Develop composition schemes that fuse your model’s output. |
| S2.3 |
Agree on the modeling process and set it up: ▪ Commission and use an experiment tracker. ▪ Commission and use a model repository. ▪ Identify and reject obviously poor models. |
| S2.4 |
Implement and commission the test environment. |
| S2.5 |
Develop a set of tests to determine the model’s functional performance. Determine the measurements to use over the test scenarios. |
| S2.6 |
Test nonfunctional features. |
| S2.7 |
Use the evaluation data to determine the model to use. Employ an explicit mechanism to use the model test/evaluation data to choose the model that used in production. Account for how component models will be combined in your design. Account for nonfunctional requirements. Account for the qualitative aspects of the model. |
| S2.8 |
Write a model delivery report. |
| S2.9 |
Determine and document model selection. |
| S2.10 |
Use the sprint backlog to locate evaluation work alongside model design, experiment tracking, candidate screening, and test-environment setup. section 7.1
