Reproducibility makes a training result traceable and repeatable: the same training inputs should produce the same model, or a model of the same quality. It is a production requirement for debugging, performance troubleshooting, and trust in decisions built on predictions. section 3.1.2 section 3.1.1
Treat training data as a fixed, versioned snapshot rather than a live dataset. A snapshot records the exact files selected by time and filtering logic; its version string lets a later run retrieve the same training data. section 2.1.3 section 2.4
A reproducible training system tracks and preserves the inputs and execution path:
- Version training datasets and fetch them by version hash or snapshot identifier. section 2.4
- Track dataset IDs and versions, training-code versions, and hyperparameters for each run. section 3.1.1
- Package training code and associate trained models with metadata and artifacts in the production workflow. section 10.2.4
- Version workflow steps and save their inputs and outputs when orchestration spans local and production environments. section 9.3.3
Rebuilding a visual-recognition experiment
Train experimental models against one versioned ImageNet dataset, vary hyperparameters, and compare metrics. Keep code in Git so each result can be traced to a code version; later, retrieve the same dataset snapshot and rerun the recorded configuration. section 10.1.1 section 2.1.3
Figure 10.7 A typical production model training setup. The workflow service manages what, when, and how training flows are run. The training service runs model training jobs. The metadata and artifacts store provides training code, stores trained models, and associates them with metadata.
A production setup separates workflow control, training execution, and metadata/artifact storage so a training flow can be rerun and its model associated with its inputs. section 10.2.4
