Query strategy

As explained in Machine Learning Algorithms in Depth

A query strategy is a criterion for selecting unlabeled training examples to be labeled in active learning.

A query strategy selects unlabeled training examples for labeling in active learning. It prioritizes examples expected to provide the most useful information. section 7.3.1

Common criteria include:

• Uncertainty sampling queries least-certain predictions: points near a binary decision boundary, low-margin or high-entropy multiclass predictions, and high-variance regression predictions. section 7.3.1
• Query by committee maintains competing models and selects candidates on which their label predictions disagree most. section 7.3.1

Logistic-regression active learner

An implementation ranks an unlabeled pool with least-confidence, max-margin, entropy, vote-entropy, or average-KL-divergence strategies, then queries a selected number of examples. A synthetic two-class logistic-regression experiment compares active selection with random sampling.

Figure 7.11 Uncertainty sampling techniques comparison. The left figure shows that active learning is better than random sampling. The right figure shows that average KL divergence is better than random sampling.

Compare uncertainty-sampling techniques with random sampling and inspect reported performance differences. section 7.3.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines uncertainty and committee criteria and implements ranking, but does not fully derive their formulas.

Examples

Uses decision-boundary points, multiclass margins and entropy, committee votes, and a synthetic logistic-regression experiment.

What it assumes

Assumes model predictions and class probabilities, while treating labeled and unlabeled pools as active-learning inputs.

What you can do after

Build it from scratch by ranking an unlabeled pool with uncertainty or committee-based criteria.

How it connects

Connects active learning with uncertainty sampling, query by committee, classification, and regression.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Related concepts

• active learningpart of · Active learning is a machine-learning technique in which a learner selectively chooses data examples for labeling or feedback to improve a model.
• query by committeeincludes · Query by committee selects examples on which a committee of models disagrees most strongly for annotation.
• uncertainty samplingincludes · Uncertainty sampling is an active-learning strategy that selects instances for which a model is most uncertain about the predicted label.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.