liveBook · Manning

Transformation provenance

As explained in LLM Customization and Fine-Tuning

Information recording the automated transformations applied to a data example.

Transformation provenance records the automated changes applied to each data example. Entries can cover PII scrubbing, format conversion, quality-gate filtering, and synthetic augmentation. For PII scrubbing, record that it occurred, the tool used, and the entity types detected and replaced; this provides evidence that personal data was removed before training. section 3.5.3

Keep the lineage chain with the dataset manifest so it travels with the data. Parent-hash links let new dataset versions extend the chain and support tracing a model checkpoint back to its original data sources. section 3.5.3

Extending a dataset lineage

When creating a v2 dataset by adding targeted synthetic examples to v1, have the v2 manifest reference the v1 SHA-256 as its parent. This extends rather than replaces the lineage chain. section 3.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Details transformation types, PII-scrubbing audit fields, manifest placement, and parent hashes across dataset versions.
Examples
Uses PII scrubbing, format conversion, quality-gate filtering, and synthetic augmentation as recorded transformations.
What you can do after
Operate lineage records to show auditors that personal data was scrubbed before training and trace checkpoints back to original data sources.
How it connects
Connects data lineage with dataset manifests, PII scrubbing, synthetic augmentation, and dataset versioning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.