concept human review in category machine learning

This is an excerpt from Manning's book Human-in-the-Loop Machine Learning.
You may not realize it, but you've probably used Active Learning before. As you learned in Chapter 1, Active Learning is the process of selecting the right data for human review. Filtering your data by keyword or some other pre-processing step is a form of Active Learning, although not a very principled one.
Finally, your camera is capturing up to 100 plants in every image. So, you have to decide how to resolve the image-level confusion with the object-level confusion. Do you prioritize human review when one object in the image is very confusing or when 100 objects are a little confusing? Do you prioritize the correct label or the accuracy of identifying the object outline? Any one of these types of error could be the most important for the problem you are addressing, so you will need to decide how to map your real-world problem to the right sampling and evaluation strategy. Therefore, even though you are automating one of the most common and repetitive tasks in human history, you need advanced Active Learning techniques to solve the problem.
Figure 7.4: Three different workflows for in-house annotation. The top one is recommended only for pilots: it doesn’t use in-house annotators at all and ignores items that are hard to annotate. The second example is the most common in industry: hard examples are redirected to experts for human review. This works fine if you have data that is fairly consistent over time. If your data is changing quickly, then the third method is recommended. In the third method, the expert annotators are looking for potential new edge cases with Diversity Sampling and Uncertainty Sampling before the main annotation process happens, and then these examples and updated guidelines are passed to the main annotation team. This is the only method to ensure that the guidelines are not trailing the actual data. The expert annotators also annotate some randomly selected items, for quality-control reasons that we will cover in the next chapter. You can combine 2 and 3, too.
![]()
![]()
![]()