An ensemble combines predictions from multiple models to make a prediction or estimate uncertainty. In active learning, disagreement among models can identify unlabeled items for human review. section 3.4.1 section 3.3.4
Ways to combine predictions for uncertainty include:
- Rank items by each model’s uncertainty, then combine their ranks when confidence scores from different models are not directly comparable. section 3.4.1
- Measure label disagreement or combine confidence values using statistics such as averages, ranges, ratios, or entropy. section 3.4.1
- For object localization, average intersection-over-union across predictions; count missing predictions as zero. section 6.1.2
Diversity helps reveal uncertainty: train models on different data or features, vary their settings, or generate repeated predictions with Monte Carlo dropout. Ensembles can require costly repeated training, while dropout offers a faster, simpler alternative. section 6.1.2 section 6.4.2
Figure 6.3 An example prediction heat map for an object, showing low variation (left) and high variation (right). The high variation is evidence of more uncertainty in the model; therefore, the right example is a good candidate for human evaluation. You can generate multiple predictions by using ensemble models, getting predictions from multiple models and changing the parameters, using a subset of features or a subset of items, or introducing random variation into your models in some other way. Within a single model, you can generate multiple predictions for a single item by using a dropout over a random selection of neurons for each prediction (known as Monte Carlo dropouts). You can also combine both methods: create an ensemble of models, and use dropouts for multiple predictions per model.
Compare low- and high-variation object predictions to see how disagreement can guide human review. section 6.1.2











