concept semantic segmentation task in category machine learning

This is an excerpt from Manning's book Human-in-the-Loop Machine Learning MEAP V09. Login to get full access to this book.

There is much current machine learning research focused on transfer learning: adapting a model from one task to another. This technique opens up some interesting possibilities for annotation strategies. For example, if your annotation task is especially time-consuming, like semantic segmentation, then you might be able to annotate orders of magnitude more data in some other way and then use that data in a model that is then adapted to the semantic segmentation task. We’ll return to this specific example later in this section.

Figure 9.15: An example of how transfer learning can be used to make a model more accurate and how it might influence the annotation strategy. Here, there are 3 other models that are feeding into a semantic segmentation model. The top example is adapting a model trained on ImageNet, which is the most common type of transfer learning. The second model is trained on 300K image-level labels for the objects that we care about. The third model uses the time-stamps of the images to train a model to predict the time of day. Because the top 3 models have all been trained on much more data than the semantic segmentation model with only 10K training items, they should have richer representations of the image that can help with the semantic segmentation task.

It is common for semantic segmentation tasks to require that all pixels receive a label, which can be problematic when annotators are uncertain about some regions or where multiple interpretations are valid. The simplest way to elicit subjectivity for semantic segmentation is to have an extra label called “uncertain”, or a similar name, that the annotator can use to indicate that they don’t know the correct label for that region. The “uncertain” region can be a separate region or you can ask the annotator to layer the “uncertain” region on top of a completed segmentation in order to know what the most likely label was, despite the confusion.