Annotation provenance records the human labeling history for each labeled example. Capture an anonymized or pseudonymized annotator identifier, annotation date, task-brief version, and agreement scores when multiple annotators review an example. section 3.5.3
These records support:
- Quality retrospectives, by helping identify whether a changed task brief introduced inconsistency in an annotation batch. section 3.5.3
- Annotator-bias analysis and honoring deletion requests from annotators who withdraw consent. section 3.5.3
Investigating an annotation batch
If a model fails on examples from one annotation batch, check the annotation dates and task-brief versions for evidence that a changed brief introduced inconsistencies. section 3.5.3
