liveBook · Manning

Dataset versioning

As explained in LLM Customization and Fine-Tuning

The practice of tracking and preserving distinct dataset versions for traceability and reproducibility.

Dataset versioning preserves the identity and provenance of training data so teams can reproduce fine-tuning runs, investigate model regressions, and support audit trails. Pair each dataset version with the training run and resulting model checkpoint. section 3.5

A minimum viable versioning system should:

  • Content-hash each dataset at creation and store the hash in a manifest that travels with the data. section 3.5
  • Log each run’s dataset SHA-256, full hyperparameter configuration, and per-category evaluation scores. section 3.5
  • Record each example’s source, annotation, and transformation provenance as it enters the pipeline. section 3.5

Trace a dataset change

When creating v2 by adding targeted synthetic examples to v1, put v1’s SHA-256 in v2’s manifest as its parent. If a model regresses, compare hashes: matching hashes point the investigation toward hyperparameters or infrastructure; differing hashes let the team inspect added, removed, or changed examples. Provenance can further distinguish a source or annotation quality problem from reduced category coverage. section 3.5.3 section 3.5.3 section 3.5

Experiment trackers such as Weights & Biases and MLflow can link hyperparameters, infrastructure, loss curves, and evaluation scores to dataset and checkpoint identifiers. section 3.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Specifies hashes, manifests, run logs, and provenance records, then applies them to regression diagnosis.
Examples
Uses a synthetic-data v2 manifest and regression checks based on hashes, provenance, and category coverage.
What you can do after
Diagnose regressions by checking dataset hashes, inspecting changed examples, and tracing data provenance.
How it connects
Links dataset versions to experiment tracking, data lineage, model checkpoints, hyperparameters, and evaluation scores.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.