concept each label in category machine learning

appears as: each label
Human-in-the-Loop Machine Learning

This is an excerpt from Manning's book Human-in-the-Loop Machine Learning.

Diversity Sampling can mean using intrinsic properties of the dataset, like the distribution of labels. For example, you might want to deliberately try to get an equal number of human annotations for each label, even though some labels are much rarer than others. Diversity Sampling can also mean ensuring that the data is representative of important external properties of the data, like ensuring that data comes from a wide variety of demographics of the people represented in the data, in order to overcome real-world bias in the data. We will cover all these variations in depth in the chapter on Diversity Sampling.

Figure 8.8: Krippendorff's alpha, calculating the overall level of agreement in a dataset to determine if it is reliable enough to use for training data. The inputs are the white areas: the Annotations (bottom right) and the Label Weights (top middle). Because we have mutually exclusive labels, this simply has each label weighted only with itself, but if we had hierarchical, ordinal or other types of data then we would enter different values as Label Weights. The top row of calculations contains the expected agreement by random chance and the bottom row of calculations calculates the actual agreement in the data. The two are used to calculate the expected agreement (pe) and actual agreement (pa) for the dataset, from which the adjusted overall agreement alpha is calculated.
  • You can aggregate multiple annotations to create a single label for each task. This allows you to create the training data for your machine learning models and to calculate the likelihood that each label is correct.