liveBook · Manning

Annotation provenance

As explained in LLM Customization and Fine-Tuning

Information recording the human labeling history and review details associated with a data example.

Annotation provenance records the human labeling history for each labeled example. Capture an anonymized or pseudonymized annotator identifier, annotation date, task-brief version, and agreement scores when multiple annotators review an example. section 3.5.3

These records support:

  • Quality retrospectives, by helping identify whether a changed task brief introduced inconsistency in an annotation batch. section 3.5.3
  • Annotator-bias analysis and honoring deletion requests from annotators who withdraw consent. section 3.5.3

Investigating an annotation batch

If a model fails on examples from one annotation batch, check the annotation dates and task-brief versions for evidence that a changed brief introduced inconsistencies. section 3.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Specifies per-example annotator identifiers, dates, task-brief versions, and review agreement scores, then connects them to operational uses.
Examples
Uses a failed annotation batch and changed task brief to illustrate investigation of inconsistent model behavior.
What you can do after
Diagnose annotation-related failures, analyze annotator bias, and honor deletion requests using per-example labeling records.
How it connects
Connects dataset lineage with source provenance, transformation provenance, quality retrospectives, annotator-bias analysis, and deletion requests.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.