liveBook · Manning

Source provenance

As explained in LLM Customization and Fine-Tuning

Information recording which source a data example came from and when it was collected or generated.

When a training example causes a problem, source provenance helps identify where it came from and when it entered the dataset. It is the first dimension of data lineage: each example should carry a machine-readable label for its originating dataset, file, or system and its collection or generation time. section 3.5.3

Record these fields for each example:

  • Source identifier: the dataset, file, or system that supplied the example. section 3.5.3
  • Collection or generation time: when the example entered the data pipeline. section 3.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the provenance label and its required fields within the broader lineage record.
Examples
Uses dataset, file, or system identifiers alongside collection or generation times as example records.
What you can do after
Apply it by attaching a machine-readable source identifier and collection or generation time to each training example.
How it connects
Connects source provenance to data lineage, annotation provenance, and transformation provenance.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.