- Methods for understanding the scores of a Machine Learning prediction so that you know when you can and cannot rely on the confidence of a prediction
- Methods for combining predictions over multiple labels into a single uncertainty score so that you have multiple ways of determining where your model is most confused
- Strategies for deciding on the right number of items to put in front of humans per iteration cycle
- Strategies for evaluating the success of uncertainty sampling to track your progress
The most common strategy that people use to make AI smarter is for the Machine Learning models to tell humans when they are uncertain about a task, and then ask the humans for the correct feedback. In general, unlabeled data that confuses a Machine Learning algorithm will be the most valuable when it is labeled and added to the training data. If the Machine Learning algorithm can already label an item with high confidence it is probably correct already.
3.1 Interpreting Uncertainty in a Machine Learning Model
3.1.1 Why look for uncertainty in your model?
3.1.2 Interpreting the scores from your model
3.1.3 “Score”, “Confidence”, and “Probability”: Do not trust the name!
3.1.4 SoftMax: converting the model output into confidences
3.2 Algorithms for Uncertainty Sampling
3.2.1 Least Confidence sampling
3.2.2 Margin of Confidence sampling
3.2.3 Ratio of Confidence sampling
3.2.4 Entropy (classification entropy)
3.2.5 A Deep Dive on Entropy
3.3 Identifying when different types of models are confused
3.3.1 What is the best activation function for Active Learning?
3.3.2 Uncertainty sampling with Logistic Regression and MaxEnt models
3.3.3 Uncertainty sampling with Support Vector Machines
3.3.4 Uncertainty sampling with Bayesian Models
3.3.5 Uncertainty sampling with Decision Trees & Random Forests
3.4 Measuring uncertainty across multiple models
3.4.1 Uncertainty sampling with Ensemble models
3.4.2 Query by Committee and Dropouts
3.5 Selecting the right number of items for human-review
3.5.1 Budget-constrained uncertainty sampling
3.5.2 Time-constrained uncertainty sampling
3.5.3 When do I stop if I’m not time or budget constrained?
3.6 Evaluating the success of uncertainty sampling
3.6.1 Do I need new test data?
3.6.2 Do I need new validation data?
3.7.1 Further Reading for Least Confidence Sampling