concept Adaptive Sampling in category machine learning

This is an excerpt from Manning's book Human-in-the-Loop Machine Learning MEAP V09. Login to get full access to this book.

Active Transfer Learning for Adaptive Sampling (ATLAS) is an exception, allowing adaptive sampling within one iteration, without having to also use clustering to ensure diversity like you learned earlier in this chapter.

Figure 5.ATLAS: Active Transfer Learning for Adaptive Sampling. Because our sampled items will later get a human label, we can assume that the model will later label those items correctly, because models are typically the most accurate on the actual items that they trained on. To begin with, validation items are predicted by the model and bucketed as “Correct” or “Incorrect” according to whether they were classified correctly or not. The last layer of the model is then retrained to predict whether items are “Correct” or “Incorrect”, effectively turning the two buckets into new labels. We then apply that to the unlabeled data, predicting whether each item will be “Correct” or “Incorrect”. You can then sample the most likely to be “Incorrect”.  Then, we can assume that those items will later get label and become part of the training data, which will be labeled correctly by a model that predicted on that same data. So, we can take those sampled items, change their label from “Incorrect” to “Correct”, and then retrain our final layer(s) on the new dataset.