concept subjective task in category machine learning
This is an excerpt from Manning's book Human-in-the-Loop Machine Learning MEAP V09.
For simple tasks, like binary labels on objective tasks, the statistics are fairly straightforward to decide which is the ‘correct’ label when different annotators disagree. But for subjective tasks, or even objective tasks with continuous data, there are no simple heuristics for deciding what the correct label should be. Think about the critical task of creating training data by putting a bounding box around every pedestrian for a self-driving car. What if two annotators have slightly different boxes? Which is the correct one? It’s not necessarily either individual box or the average of the two boxes. In fact, the best way to resolve this problem is with Machine Learning itself.
Therefore, in many cases where there is ambiguity about whether a label is valid or not for a subjective task, you will want to find another annotator (possibly an expert) that you can trust to understand the diversity of possible responses.
Figure 9.7: An example where a subjective task has an additional question that is unambiguous. This allows easier quality control by assuming that if a person gets the unambiguous question correct, then their subjective judgment is also correct and not an error.
In Figure 9.7, we are asking an additional question about whether the sky can be seen in the message. Unlike the object type, this should be unambiguous in almost all cases: either the sky is visible or not. Therefore, we can easily test whether people are getting the product question correct by embedded known answers for some questions and/or by looking for agreement between annotators, using the earlier techniques in this chapter. We then assume that the people are equally accurate for the subjective task.
When using this method, it relies on the assumption that accuracy for the simpler task will strongly correlate with accuracy for the subjective task. This will be more or less true depending on your data. As a general principle, the closer the question is to the relevant content, the closer this correlation should be. In our example, we are asking about the context of the object, so the accuracy should be highly correlated.