Documentation records the work and decisions needed to deliver and maintain an ML system. Working code demonstrates progress, but it does not tell a support team how to operate the system or fix it when something goes wrong. Keep documents useful to the people who will need them, including the developer returning to a decision weeks later. section 4.3.5
Build the record as the project proceeds. At each sprint checkpoint, check the documentation requirements alongside completed tasks. The resulting archive lets team members share and reuse information and later supports production maintenance and governance. section 1.4
Make documentation an explicit deliverable at two stages:
- During modeling and testing, record feature and model designs, the details needed to reproduce models, experimental results, and defects. Keep those records current when new data or constraints require rebuilt pipelines and repeated tests. section 8.4 section 7.4.1
- Before production handover, prepare a production-team organization chart, run book, technical overview, and troubleshooting guide. Agree on governance and handover arrangements as part of the support work. section 9.5.2 section 9.5
Turn an alert into a support action
For each error state exposed by logging and monitoring, give support staff a run-book entry. An entry might direct them to escalate a data error or restart an application server; it should help them raise the right ticket and inform users what is happening. section 9.5.2
Table 8.3 Sprint 2 checklist (view table figure)
| Ticket # |
Item |
Notes |
|---|---|---|
| S2.1 |
Feature engineering is implemented. |
Features and design information are recorded. |
| S2.2 |
Models to be used explicitly are designed. |
Model design documented. |
| S2.3 |
Modelling process undertaken and models developed. |
Model repository used; models identified and captured along with all details required for reproduction. |
| S2.4 |
Model performance in dev was properly assessed by the team, and the experimental results are recorded. |
Results of experiments are available for inspection. |
| S2.5 |
Model issues discovered in dev are recorded. |
Defects will often be noticed in development; check that these were documented and that the documentation is available. |
| S2.6 |
Test environment was commissioned. |
The test environment is set up, and the required data sources for testing are available. |
| S2.7 |
Appropriate tests are designed. |
The tests that determine if the models are fit for purpose agreed on and documented. |
| S2.8 |
Test data has been gathered. |
Tests run in the test environment, and results collected and available for evaluation. |
| S2.9 |
Model selection is documented. |
Model selection methods are documented, and the decisions that were made when using them and the testing data are recorded. Qualitative factors in the decisions should be recorded and agreed on. |
Use the post-modeling checklist to verify that designs, reproducibility details, experiment results, and discovered issues have been recorded before moving on. section 8.4
