Provenance, or lineage, connects an ML artifact to its origins and the steps that produced it. A model’s weights and its supporting code are separate entities; tracking both becomes essential as datasets, configurations, and deployed versions change. Without lineage, correcting bad data may mean rerunning extraction scripts instead of recreating a dataset with a query. section 2.1 section 2.1.1
Keep the links needed to answer production questions:
- For data, record when, where, how, and why it was collected. Track raw data alongside intermediate, augmented, and annotated versions; generate datasets through a versioned ETL pipeline. section 2.1.1
- For training, tie each model to a dataset version and log its configuration, code, checkpoints, metrics, and other artifacts with the experiment run. Keep model, code, configuration, and data as distinct versioned components. section 2.1.3 section 9.4
- For deployment, retain the model version and its transitions through staging and production. A registry can connect registered versions to runs, control promotion, support audits, and make rollback possible. section 9.3.5 section 2.1.5
Changing a car-price dataset
A used-car price estimator needs data representative of passenger cars on its target market. If the first dataset proves unsuitable, collection metadata and a versioned ETL pipeline make it easier to revise the selection and reproduce the dataset. Tying a trained model to that dataset version then makes the effect of the revision traceable. section 2.1.1 section 2.1.3
Figure 2.5 A view of a retraining pipeline using the modular codebase concept. This approach of keeping the model, code, configuration files, and data as distinct versioned components with lineage links ensures that the process remains flexible, fast, and adaptable while enabling experimentation, debugging, and iterative development.
The retraining-pipeline diagram shows why data, code, configuration, and models remain separate versioned components with lineage links. section 2.1.3
In an automated workflow, lineage is useful only if the training and serving paths preserve it. Choose data access that supports version control, use experiment tracking to retain training results, and register promising models before promotion. MLflow can provide model tracking and a registry; BentoML can serve models from that registry. Well-kept links shorten production debugging and help teams identify which version to restore. section 8.2 section 9.3.5 section 9.4 section 10.9
