Uncertainty sampling is an active-learning query strategy: instead of labeling training examples at random, select unlabeled examples whose predicted labels are most uncertain under the current model. The aim is to spend labeling effort where an answer is most informative. section 7.3.1 section 7.8
For a classifier that assigns probabilities to K classes, uncertainty can be ranked in several ways:
- Least confidence: prefer the example with the smallest probability assigned to its most probable class. In binary logistic regression, this favors predictions near ½ and thus near the decision boundary. section 7.3.1
- Smallest margin: prefer the smallest difference between the probabilities of the two most probable classes. Unlike least confidence, this uses information about both leading predictions, but still ignores the remaining classes. section 7.3.1
- Highest entropy: prefer the most uncertain overall predicted class distribution. For regression, use the greatest predicted output variance instead. section 7.3.1
To use a pool-based strategy, score the unlabeled candidates with the current model, rank them by the chosen criterion, and request labels for the selected queries. Ranking the entire pool can be costly for very large datasets; stream-based sampling instead scans examples sequentially and evaluates each query decision individually. Query by committee is a different strategy: it selects examples on which competing models disagree. section 7.3 section 7.3.1
Logistic regression with a fixed label budget
On a balanced synthetic binary-classification dataset, logistic regression trained using 30 randomly sampled labels reaches 90% accuracy on held-out data. Using 30 entropy-selected queries instead reaches 92.5%. The comparison illustrates the possible benefit of choosing labels by uncertainty; it does not establish that uncertainty sampling always improves accuracy. section 7.3
Listing 7.5 Active learner class
from __future__ import unicode_literals, division
from scipy.sparse import csc_matrix, vstack
from scipy.stats import entropy
from collections import Counter
import numpy as np
class ActiveLearner(object):
uncertainty_sampling_frameworks = [
'entropy', #1
'max_margin', #1
'least_confident', #1
]
query_by_committee_frameworks = [
'vote_entropy',
'average_kl_divergence', #2
] #2
def __init__(self, strategy='least_confident'):
self.strategy = strategy
def rank(self, clf, X_unlabeled, num_queries=None):
if num_queries == None:
num_queries = X_unlabeled.shape[0]
elif type(num_queries) == float:
num_queries = int(num_queries * X_unlabeled.shape[0])
if self.strategy in self.uncertainty_sampling_frameworks:
scores = self.uncertainty_sampling(clf, X_unlabeled)
elif self.strategy in self.query_by_committee_frameworks:
scores = self.query_by_committee(clf, X_unlabeled)
else:
raise ValueError("this strategy is not implemented.")
rankings = np.argsort(-scores)[:num_queries]
return rankings
def uncertainty_sampling(self, clf, X_unlabeled):
probs = clf.predict_proba(X_unlabeled)
if self.strategy == 'least_confident':
return 1 - np.amax(probs, axis=1) #3
elif self.strategy == 'max_margin':
margin = np.partition(-probs, 1, axis=1)
return -np.abs(margin[:,0] - margin[:, 1]) #4
elif self.strategy == 'entropy':
return np.apply_along_axis(entropy, 1,
probs) #5
def query_by_committee(self, clf, X_unlabeled):
num_classes = len(clf[0].classes_)
C = len(clf)
preds = []
if self.strategy == 'vote_entropy':
for model in clf:
y_out = map(int, model.predict(X_unlabeled))
preds.append(np.eye(num_classes)[y_out])
votes = np.apply_along_axis(np.sum, 0, np.stack(preds)) / C
return np.apply_along_axis(entropy, 1,
votes) #6
elif self.strategy == 'average_kl_divergence':
for model in clf:
preds.append(model.predict_proba(X_unlabeled))
consensus = np.mean(np.stack(preds), axis=0)
divergence = []
for y_out in preds:
divergence.append(entropy(consensus.T, y_out.T))
return np.apply_along_axis(np.mean, 0, np
.stack(divergence)) #7
The active-learner implementation ranks unlabeled examples using selectable least-confidence, margin, or entropy strategies. section 7.3.1
