Accuracy measures how closely an annotator’s work matches a reference. For a labeling task with known answers, it is the fraction of answers the annotator gets right. For a subjective task, accuracy can instead measure how well an annotator predicts the distribution of responses across other annotators; it does not establish that any one response is correct. section 8.1.1 section 9.1.3
Estimating a response distribution
Suppose an annotator expects 65% of people to label an image “Cyclist,” but 40% actually do. The estimate is off by 0.25. In the scoring method used here, half the possible estimates are closer to the observed fraction; adjusting for the 0.2 precision imposed by five annotators gives a score of 0.5 × (1 − 0.2) + 0.2 = 0.6, or 60%. Scores across estimates can be combined for a task and averaged across subjective tasks to obtain an overall annotator score. section 9.1.3 section 8.1
Figure 9.5 Calculating the accuracy of each annotator’s estimate as the adjusted score and then averaging those scores to get a per-annotator score for this task. Cameron is 80% accurate in estimating how close the expected distribution was to the actual distribution. Evan is the most accurate, with a score of 97%, and Blake is the least accurate, with a score of 73%.
Compare adjusted distribution-estimate scores across annotators for one task. section 9.1.3
When known answers are available, use a random or representative ground-truth sample to assess individual annotators. For an active-learning batch, a sample from that batch helps assess accuracy on the items being labeled. Ground-truth errors can make both the accuracy metric and the resulting training data misleading. section 8.1
Raw agreement with ground truth can be adjusted for chance. The result depends on the baseline: guessing uniformly, always choosing the most frequent label, or guessing according to label frequencies. A score below zero after adjustment indicates performance below the chosen baseline. section 8.1.1
Accuracy also supports decisions about which annotations to trust. Compare agreement-uncertainty scores with ground-truth accuracy to choose a threshold; alternatively, test how different thresholds affect downstream model accuracy. Revisit the threshold as training data grows. section 8.3.5
The appropriate measure depends on the task. Continuous responses can be scored against an acceptable range, while bounding boxes can be compared with ground truth using intersection over union. Keep scores for different task types separate when their metrics are not directly comparable. section 10.1.1 section 10.2.1 section 10.2.3

































