Testing is a systematic, discrete evaluation of models that produces comprehensible evidence for model selection. It turns competing model outcomes into choices that can be justified to users, explained to stakeholders, and audited by regulators. section 8.0
Treat testing as a project process rather than a single score. Structure the environment, migrate code and artifacts, measure model properties, and choose suitable offline or online tests. Document and reproduce the process; apply only the tests the project needs. section 8.0 section 8.1
Use testing to control delivery risk through explicit project actions:
- Record model designs, features, experiments, development performance, and discovered defects in the model repository and project documentation. section 8.4
- Commission a test environment with the required data sources before evaluating candidates. section 8.4
- Revisit tests when integration exposes new constraints, data sources, or data errors; rebuild pipelines, rerun tests, and update documentation transparently. section 8.4
- Review testing evidence when model performance drifts, because inadequate testing may reveal weaknesses requiring repair or retrofit. section 10.2.2
Testing after model development
A team finishes development with several candidate models. It records their designs and experiments, commissions a test environment, runs appropriate offline or online evaluations, captures defects, and preserves the evidence needed for deliberate selection. If application integration reveals a new data problem, the team rebuilds the affected pipeline and reruns the tests. section 8.1 section 8.4
Table 8.3 Sprint 2 checklist (view table figure)
| Ticket # |
Item |
Notes |
|---|---|---|
| S2.1 |
Feature engineering is implemented. |
Features and design information are recorded. |
| S2.2 |
Models to be used explicitly are designed. |
Model design documented. |
| S2.3 |
Modelling process undertaken and models developed. |
Model repository used; models identified and captured along with all details required for reproduction. |
| S2.4 |
Model performance in dev was properly assessed by the team, and the experimental results are recorded. |
Results of experiments are available for inspection. |
| S2.5 |
Model issues discovered in dev are recorded. |
Defects will often be noticed in development; check that these were documented and that the documentation is available. |
| S2.6 |
Test environment was commissioned. |
The test environment is set up, and the required data sources for testing are available. |
| S2.7 |
Appropriate tests are designed. |
The tests that determine if the models are fit for purpose agreed on and documented. |
| S2.8 |
Test data has been gathered. |
Tests run in the test environment, and results collected and available for evaluation. |
| S2.9 |
Model selection is documented. |
Model selection methods are documented, and the decisions that were made when using them and the testing data are recorded. Qualitative factors in the decisions should be recorded and agreed on. |
Use the Sprint 2 checklist to verify feature and model records, reproducible experiments, documented defects, a commissioned test environment, and appropriate tests before application integration. section 8.4
