Model development turns data, application requirements, and known constraints into models that can be tested and used. It is not just training: design choices, records of experiments, evaluation, and decisions to reject weak models all affect whether a result is useful. Even a one-click model needs a disciplined process before and after creation. section 6.4 section 7.4
Start only after the team understands the data and has documented business, data, and customer-system constraints. Prepare the data pipelines, survey data quality and coverage, agree on model performance requirements, and establish a repository for model artifacts. These tasks reduce uncertainty before the modeling sprint. section 5.1 section 6.4
Run development as a documented backlog rather than an unrecorded sequence of trials:
- Plan features, implement a feature-engineering pipeline, and decide whether data augmentation is needed. section 7.1
- Document the model design: account for design constraints, divide the task where appropriate, choose components suited to the data, and decide how their outputs will be combined. section 7.1
- Agree on an experiment process, use an experiment tracker and model repository, and reject obviously poor models early. section 7.1
- Commission a test environment and develop tests for functional performance. section 7.1
Keep enough design and experiment records to rebuild models and avoid repeating discarded work. Guard evaluation data carefully: repeated optimization against validation results can reward quirks that do not hold in production. Findings may also send the team back to data exploration or modeling; integration can expose a model property that requires redesign. section 7.4.1 section 1.4
A misleading time-series result
A smart-building model is validated on observations from a hot summer, while its test set also comes from that summer. A signal from future cases can make test performance look convincing without showing that the model will perform as well in production. Revisit how training, validation, and test cases were chosen before accepting the result. section 7.4.1
Table 7.2 Backlog for Sprint 2: modelling and evaluation (view table figure)
| Task # |
Item |
|---|---|
| S2.1 |
Create a feature engineering plan, then share and review it with team. Implement a feature-engineering pipeline. Design and add data augmentation. |
| S2.2 |
Create the model’s design and document it. Consider the forces acting on the model’s design. Decide on the decomposition of tasks to create your overall design. Choose component parts based on the required inductive biases given your data. Develop composition schemes that fuse your model’s output. |
| S2.3 |
Agree on the modeling process and set it up: ▪ Commission and use an experiment tracker. ▪ Commission and use a model repository. ▪ Identify and reject obviously poor models. |
| S2.4 |
Implement and commission the test environment. |
| S2.5 |
Develop a set of tests to determine the model’s functional performance. Determine the measurements to use over the test scenarios. |
| S2.6 |
Test nonfunctional features. |
| S2.7 |
Use the evaluation data to determine the model to use. Employ an explicit mechanism to use the model test/evaluation data to choose the model that used in production. Account for how component models will be combined in your design. Account for nonfunctional requirements. Account for the qualitative aspects of the model. |
| S2.8 |
Write a model delivery report. |
| S2.9 |
Determine and document model selection. |
| S2.10 |
Use the sprint 2 backlog to turn feature engineering, model design, experiment tracking, and testing into assigned, reviewable work. section 7.1
