Provenance records where data comes from, how it changes, which features depend on it, and how it moves through processing systems. In ML pipelines, it supports reproducibility and helps investigators trace data-quality problems to complex transformations. section 11.4.2 section 14.3.6
Useful provenance practices include:
- Record feature versions alongside the timestamps and feature contents in each dataset. section 11.4.2
- Separate changed calculation methods or data sources into new feature versions rather than mixing old and new values. section 11.4.2
- Trace raw inputs, postprocessed feature values, and pipeline transformations when investigating data-quality failures. section 14.3.6
Changing a feature calculation
If an engineer changes a feature’s calculation principle or data source, treating it as the same feature can mix incompatible values. Writing a new version and backfilling historical values preserves consistency and allows an older model’s results to be reproduced. section 11.4.2
Figure 11.15 A graph of feature dependencies
A feature-dependency graph makes relationships between derived features and their upstream calculations visible for tracing and reuse. section 11.4.2
