concept annotator in category machine learning

This is an excerpt from Manning's book Human-in-the-Loop Machine Learning.
However, when you are dealing with more complicated interfaces, the conventions become more complicated. Imagine you are asking people to put polygons around certain objects in an image, which is a common use case for autonomous vehicle companies. What modalities would an annotator expect? Would they expect free-hand, lines, paint-brushes, smart selection by color/region, and other selection tools? If people are accustomed to working on images in programs like Adobe PhotoShop, then they might be expecting the same functionality for annotating images for Machine Learning. Just as you are building on and constrained by people’s expectations for web forms, you are constrained by their expectations for selecting and editing images. Unfortunately, those expectations might require 100s of hours of coding to build if you are offering fully-featured interfaces.
For anyone who is undertaking repetitive tasks like creating training data, moving a mouse is inefficient and should be avoided if possible. If the entire annotation process can happen on a keyboard, including the annotation itself and any form submissions or navigations, then the rhythm of the annotators will be greatly improved. If you have to include a mouse, you should be getting rich annotations to make up for the slower inputs.
Figure 8.1: The flow of information for annotation. In addition to taking data sampled according to our current Active Learning strategies, we also sample a random or representative set of data and from within the annotation itself. Sampling Random/Representative data lets us calculate the accuracy of the annotator in a way that is easiest to determine their reliability across different datasets and whether they are candidates for promotion to experts. Sampling within the current Active Learning batch allows us to calculate the accuracy for this particular dataset. Sampling during the annotation process finds items that are most useful for annotation guidelines and for adjudication by experts.
![]()
The most reliable annotators. The annotators with high agreement are likely to be the most accurate for your task, so identifying these people for potential reward and promotion is helpful. Collaboration between annotators. Do any annotators agree with each other near perfectly? They might be sharing notes innocently because they sit near each other, in which case you need to remove those responses from any calculations of agreement that assumes independence. On the other hand, this might be evidence that a bot that is duplicating one person’s work to wrongly get paid twice. Regardless of the underlying cause, it is helpful to know when two sets of answers are really only one that has been repeated. An annotator’s consistency over time. If you give the same task to the same person at different times, do they give the same result? Known as intra-annotator agreement, this can be evidence that an annotator is not paying attention; that your task has ordering effects; and/or that the task is inherently subjective. Creating examples for the instructions. You can assume that items with high agreement from a large number of annotators are correct, and let these automatically become examples in the guidelines for new annotators. You run the risk of some errors still getting through and propagating, and also that only easier tasks get through with higher agreement, so this strategy should not be used as the only one for creating ground-truth data.