A data source is a system, archive, experiment, survey, or other origin that provides data for a downstream analysis or model. Treating it as a delivery concern means documenting where data lives, what the system does, how it was obtained, and whether its contents represent the intended population. section 4.5.4 section 2.4
Turn data-source discovery into an explicit project task:
- Create a discovery ticket and obtain a representative sample before committing to full extraction. section 2.4
- Record statistical properties, scale, speed, history, location, infrastructure, and system purpose. section 2.4
- Identify cleaning, quality, access, contractual, security, and supplementation work exposed by the sample. section 2.4
- Track collection events and time periods so operational snapshots are not silently treated as one homogeneous asset. section 4.5.4 section 5.4.1
Adding sources can expand the available data picture, but fusion can also introduce bias when controlled studies, observational data, different populations, or different periods are combined. Preserve provenance and revisit source assumptions when testing or integration reveals defects. section 5.4.1 section 8.4
Representative sample before full access
A client may identify one training data set, but a sample can reveal additional relevant systems, missing history, quality problems, or the need for open or commercial data. Full extraction may wait until funding, contracts, or security access are available. section 2.4
Table 8.3 Sprint 2 checklist (view table figure)
| Ticket # |
Item |
Notes |
|---|---|---|
| S2.1 |
Feature engineering is implemented. |
Features and design information are recorded. |
| S2.2 |
Models to be used explicitly are designed. |
Model design documented. |
| S2.3 |
Modelling process undertaken and models developed. |
Model repository used; models identified and captured along with all details required for reproduction. |
| S2.4 |
Model performance in dev was properly assessed by the team, and the experimental results are recorded. |
Results of experiments are available for inspection. |
| S2.5 |
Model issues discovered in dev are recorded. |
Defects will often be noticed in development; check that these were documented and that the documentation is available. |
| S2.6 |
Test environment was commissioned. |
The test environment is set up, and the required data sources for testing are available. |
| S2.7 |
Appropriate tests are designed. |
The tests that determine if the models are fit for purpose agreed on and documented. |
| S2.8 |
Test data has been gathered. |
Tests run in the test environment, and results collected and available for evaluation. |
| S2.9 |
Model selection is documented. |
Model selection methods are documented, and the decisions that were made when using them and the testing data are recorded. Qualitative factors in the decisions should be recorded and agreed on. |
Use the checklist to confirm that testing has the required data sources, recorded experiments, documented defects, and reproducible model artifacts. section 8.4
