Experiment tracking systematically records the parameters, inputs, outputs, and performance of model runs. Establish it as part of the modelling process alongside a model repository, because even one-click model creation requires disciplined controls. section 7.4 section 7.4.2
Set up tracking to preserve the evidence needed to compare runs and explain their results:
- Record algorithm versions, hyperparameters, training-set or pipeline versions, test-set size, start time, duration, and performance metrics. section 7.4.2
- Push parameters and results from modelling code into a tracking database after every run, then retrieve them through a GUI or command-line interface. section 7.4.2
- Use unique model identifiers to connect binaries, weights, formulas, datasets, experiment results, and the corresponding model repository entry. section 7.4.2
Explaining performance variation
A sequence of linear-model runs records penalty choice, l1_ratio, test-set size, and AUC. Adding testSize after unexplained performance differences makes the comparison interpretable; a larger test set also coincides with a changed penalty function. section 7.4.2
Table 7.3 Example of statistics that might be recorded by an experiment tracking system (view table figure)
| Start time |
Duration |
l1_ratio |
Penalty |
testSize |
auc_test |
|---|---|---|---|---|---|
| 15/02/2022 10:43 |
1.9 s |
0.5 |
Elastic net |
0.5 |
0.93916314 |
| 15/02/2022 10:42 |
2.1 s |
0.5 |
l2 |
0.1 |
0.94286043 |
| 15/02/2022 10:38 |
2.1 s |
0.5 |
l2 |
0.1 |
0.94286043 |
| 15/02/2022 10:34 |
2.4 s |
0.5 |
l2 |
0.94286043 |
|
| 15/02/2022 10:00 |
2.0 s |
0.1 |
l2 |
0.94286043 |
|
| 15/02/2022 09:59 |
1.9 s |
0.5 |
Elastic net |
0.93916314 |
|
| 15/02/2022 09:58 |
2.3 s |
0.1 |
l2 |
0.94505654 |
Compare run-level timestamps, durations, hyperparameters, test-set sizes, penalties, and AUC values in the experiment statistics table. section 7.4.2
