Query by committee (QBC) is an active-learning strategy for deciding which unlabeled example to send for annotation. Instead of asking where one model is least certain, it asks where several models disagree most. The aim is to obtain labels that rule out competing hypotheses while using fewer labeled examples. section 7.3.1 section 7.0
Let L be the current labeled set and C = {θ₁, …, θₘ} a committee of models trained on L. A query proceeds as follows:
- Have each committee member vote on the label of each unlabeled candidate. section 7.3.1
- Rank candidates by disagreement among those votes. Vote entropy and average KL divergence are available QBC strategies. section 7.3.1
- Request labels for the highest-ranked candidates, then use the enlarged labeled set for the next selection. section 7.3.1
The distinction from uncertainty sampling is whose uncertainty matters. Uncertainty sampling ranks examples using one model’s confidence, margin, entropy, or prediction variance. QBC maintains competing models and ranks examples by their disagreement. An example all members classify alike offers less opportunity to distinguish their hypotheses than one that splits their votes. section 7.3.1
Listing 7.5 Active learner class
from __future__ import unicode_literals, division
from scipy.sparse import csc_matrix, vstack
from scipy.stats import entropy
from collections import Counter
import numpy as np
class ActiveLearner(object):
uncertainty_sampling_frameworks = [
'entropy', #1
'max_margin', #1
'least_confident', #1
]
query_by_committee_frameworks = [
'vote_entropy',
'average_kl_divergence', #2
] #2
def __init__(self, strategy='least_confident'):
self.strategy = strategy
def rank(self, clf, X_unlabeled, num_queries=None):
if num_queries == None:
num_queries = X_unlabeled.shape[0]
elif type(num_queries) == float:
num_queries = int(num_queries * X_unlabeled.shape[0])
if self.strategy in self.uncertainty_sampling_frameworks:
scores = self.uncertainty_sampling(clf, X_unlabeled)
elif self.strategy in self.query_by_committee_frameworks:
scores = self.query_by_committee(clf, X_unlabeled)
else:
raise ValueError("this strategy is not implemented.")
rankings = np.argsort(-scores)[:num_queries]
return rankings
def uncertainty_sampling(self, clf, X_unlabeled):
probs = clf.predict_proba(X_unlabeled)
if self.strategy == 'least_confident':
return 1 - np.amax(probs, axis=1) #3
elif self.strategy == 'max_margin':
margin = np.partition(-probs, 1, axis=1)
return -np.abs(margin[:,0] - margin[:, 1]) #4
elif self.strategy == 'entropy':
return np.apply_along_axis(entropy, 1,
probs) #5
def query_by_committee(self, clf, X_unlabeled):
num_classes = len(clf[0].classes_)
C = len(clf)
preds = []
if self.strategy == 'vote_entropy':
for model in clf:
y_out = map(int, model.predict(X_unlabeled))
preds.append(np.eye(num_classes)[y_out])
votes = np.apply_along_axis(np.sum, 0, np.stack(preds)) / C
return np.apply_along_axis(entropy, 1,
votes) #6
elif self.strategy == 'average_kl_divergence':
for model in clf:
preds.append(model.predict_proba(X_unlabeled))
consensus = np.mean(np.stack(preds), axis=0)
divergence = []
for y_out in preds:
divergence.append(entropy(consensus.T, y_out.T))
return np.apply_along_axis(np.mean, 0, np
.stack(divergence)) #7
The ActiveLearner listing names vote_entropy and average_kl_divergence as committee strategies and exposes a rank method for selecting a number or fraction of unlabeled candidates. section 7.3.1
Figure 7.11 Uncertainty sampling techniques comparison. The left figure shows that active learning is better than random sampling. The right figure shows that average KL divergence is better than random sampling.
The comparison figure reports average KL divergence performing better than random sampling in its illustrated active-learning comparison. section 7.3.1
