liveBook · Manning

Ensemble

As explained in Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

A method that combines or compares predictions from multiple models to support a prediction or decision.

An ensemble combines predictions from multiple models to make a prediction or estimate uncertainty. In active learning, disagreement among models can identify unlabeled items for human review. section 3.4.1 section 3.3.4

Ways to combine predictions for uncertainty include:

  • Rank items by each model’s uncertainty, then combine their ranks when confidence scores from different models are not directly comparable. section 3.4.1
  • Measure label disagreement or combine confidence values using statistics such as averages, ranges, ratios, or entropy. section 3.4.1
  • For object localization, average intersection-over-union across predictions; count missing predictions as zero. section 6.1.2

Diversity helps reveal uncertainty: train models on different data or features, vary their settings, or generate repeated predictions with Monte Carlo dropout. Ensembles can require costly repeated training, while dropout offers a faster, simpler alternative. section 6.1.2 section 6.4.2

Figure 6.3 An example prediction heat map for an object, showing low variation (left) and high variation (right). The high variation is evidence of more uncertainty in the model; therefore, the right example is a good candidate for human evaluation. You can generate multiple predictions by using ensemble models, getting predictions from multiple models and changing the parameters, using a subset of features or a subset of items, or introducing random variation into your models in some other way. Within a single model, you can generate multiple predictions for a single item by using a dropout over a random selection of neurons for each prediction (known as Monte Carlo dropouts). You can also combine both methods: create an ensemble of models, and use dropouts for multiple predictions per model.

Compare low- and high-variation object predictions to see how disagreement can guide human review. section 6.1.2

How Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI covers it

How deep it goes
Details rank aggregation, confidence statistics, model-diversity strategies, and IoU handling for missing localization predictions.
Examples
Demonstrates random forests, neural-model committees, object-detection localization, and uncertainty estimation for language generation.
What you can do after
Apply model disagreement or aggregated confidence to rank unlabeled items for human review.
How it connects
Connects ensembles with uncertainty sampling, active learning, random forests, Query by Committee, Monte Carlo dropout, and object detection.

Also explained in

Compare how 12 books explain ensemble →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. The section links point to the parts of the book it draws on.