concept unlabeled datum in category machine learning

appears as: unlabeled data, unlabeled data
Human-in-the-Loop Machine Learning

This is an excerpt from Manning's book Human-in-the-Loop Machine Learning.

Other types of Diversity Sampling, like Representative Sampling, are explicitly trying to find the unlabeled items that most look like the unlabeled data, compared to the training data. For example, Representative Sampling might find unlabeled items in text documents that have words that are really common in the unlabeled data but aren’t yet in the training data. For this reason, it is a good method to implement when you know that the data is changing over time.

These fields are enough information for you to build your first model. You will see that the unlabeled data in the example does not yet have a “label”, “sampling strategy” or “confidence”, for obvious reasons.

def get_low_conf_unlabeled(model, unlabeled_data, number=80, limit=10000):
    confidences = []
    if limit == -1: # we're predicting confidence on *everything* this will take a while
       print("Get confidences for unlabeled data (this might take a while)")
    else:
       # only apply the model to a limited number of items
       shuffle(unlabeled_data)
       unlabeled_data = unlabeled_data[:limit]
   
    with torch.no_grad():
        for item in unlabeled_data:
            textid = item[0]
            if textid in already_labeled:
                continue
 
            text = item[1]
 
            feature_vector = make_feature_vector(text.split(), feature_index)
            log_probs = model(feature_vector)
 
            # get confidence that it is related
            prob_related = math.exp(log_probs.data.tolist()[0][1])    #A
 
            if prob_related < 0.5:
                confidence = 1 - prob_related
            else:
                confidence = prob_related
 
            item[3] = "low confidence"
            item[4] = confidence
            confidences.append(item)
 
    confidences.sort(key=lambda x: x[4])    #B
    return confidences[:number:]
 
 
def get_random_items(unlabeled_data, number = 10):
    shuffle(unlabeled_data)
 
    random_items = []
    for item in unlabeled_data:
        textid = item[0]
        if textid in already_labeled:
            continue
        random_items.append(item)
        if len(random_items) >= number:
            break
 
    return random_items
 
 
def get_outliers(training_data, unlabeled_data, number=10):
    """Get outliers from unlabeled data in training data
    Returns number outliers
   
    An outlier is defined as the percent of words in an item in
    unlabeled_data that do not exist in training_data
    """
    outliers = []
 
    total_feature_counts = defaultdict(lambda: 0)
   
    for item in training_data:
        text = item[1]
        features = text.split()
 
        for feature in features:
            total_feature_counts[feature] += 1    #C
               
    while(len(outliers) < number):
        top_outlier = []
        top_match = float("inf")
 
        for item in unlabeled_data:
            textid = item[0]
            if textid in already_labeled:
                continue
 
            text = item[1]
            features = text.split()
            total_matches = 1 # start at 1 for slight smoothing
            for feature in features:
                if feature in total_feature_counts:
                    total_matches += total_feature_counts[feature]    #D
 
            ave_matches = total_matches / len(features)
            if ave_matches < top_match:
                top_match = ave_matches
                top_outlier = item
 
        # add this outlier to list and update what is 'labeled',
        # assuming this new outlier will get a label
        top_outlier[3] = "outlier"
        outliers.append(top_outlier)
        text = top_outlier[1]
        features = text.split()
        for feature in features:
            total_feature_counts[feature] += 1    #E
 
    return outliers
  
Figure 5.RATL: Active Transfer Learning for Representative Sampling. We can build a model to sample the items that are most unlike the current training data. To begin with, we take validation data from the same distribution as the training data and give it a "Training" label. We then take unlabeled data from our target domain and give it a "Application" label. We train a new output layer to predict the Training/Application labels, giving it access to all layers of the model. We then apply that to the unlabeled data (ignoring the unlabeled items that we just trained on) and sample the items that are most confidently predicted as "Application".