This introduction explains how human feedback, selective annotation, and existing models support human-in-the-loop machine learning.
1.1 The basic principles of human-in-the-loop machine learning · 1.2 Introducing annotation · 1.2.1 Simple and more complicated annotation strategies · 1.2.2 Plugging the gap in data science knowledge · 1.2.3 Quality human annotation: Why is it hard? · 1.3 Introducing active learning: Improving the speed and reducing the cost of training data · 1.3.1 Three broad active learning sampling strategies: Uncertainty, diversity, and random · 1.3.2 What is a random selection of evaluation data? · 1.3.3 When to use active learning · 1.4 Machine learning and human–computer interaction · 1.4.1 User interfaces: How do you create training data? · 1.4.2 Priming: What can influence human perception? · 1.4.3 The pros and cons of creating labels by evaluating machine learning predictions · 1.4.4 Basic principles for designing annotation interfaces · 1.5 Machine-learning-assisted humans vs. human-assisted machine learning · 1.6 Transfer learning to kick-start your models · 1.6.1 Transfer learning in computer vision · 1.6.2 Transfer learning in NLP · 1.7 What to expect in this text · 1.8 Summary
The chapter introduction advocates starting with a simple human-in-the-loop machine learning system and improving it through feedback.
2.1 Beyond hacktive learning: Your first active learning algorithm · 2.2 The architecture of your first system · 2.3 Interpreting model predictions and data to support active learning · 2.3.1 Confidence ranking · 2.3.2 Identifying outliers · 2.3.3 What to expect as you iterate · 2.4 Building an interface to get human labels · 2.4.1 A simple interface for labeling text · 2.4.2 Managing machine learning data · 2.5 Deploying your first human-in-the-loop machine learning system · 2.5.1 Always get your evaluation data first · 2.5.2 Every data point gets a chance · 2.5.3 Select the right strategies for your data · 2.5.4 Retrain the model and iterate · 2.6 Summary
This introduction explains why uncertain predictions are candidates for human labeling and why the choice of uncertainty measure matters.
3.1 Interpreting uncertainty in a machine learning model · 3.1.1 Why look for uncertainty in your model? · 3.1.2 Softmax and probability distributions · 3.1.3 Interpreting the success of active learning · 3.2 Algorithms for uncertainty sampling · 3.2.1 Least confidence sampling · 3.2.2 Margin of confidence sampling · 3.2.3 Ratio sampling · 3.2.4 Entropy (classification entropy) · 3.2.5 A deep dive on entropy · 3.3 Identifying when different types of models are confused · 3.3.1 Uncertainty sampling with logistic regression and MaxEnt models · 3.3.2 Uncertainty sampling with SVMs · 3.3.3 Uncertainty sampling with Bayesian models · 3.3.4 Uncertainty sampling with decision trees and random forests · 3.4 Measuring uncertainty across multiple predictions · 3.4.1 Uncertainty sampling with ensemble models · 3.4.2 Query by Committee and dropouts · 3.4.3 The difference between aleatoric and epistemic uncertainty · 3.4.4 Multilabeled and continuous value classification · 3.5 Selecting the right number of items for human review · 3.5.1 Budget-constrained uncertainty sampling · 3.5.2 Time-constrained uncertainty sampling · 3.5.3 When do I stop if I’m not time- or budget-constrained? · 3.6 Evaluating the success of active learning · 3.6.1 Do I need new test data? · 3.6.2 Do I need new validation data? · 3.7 Uncertainty sampling cheat sheet · 3.8 Further reading · 3.8.1 Further reading for least confidence sampling · 3.8.2 Further reading for margin of confidence sampling · 3.8.3 Further reading for ratio of confidence sampling · 3.8.4 Further reading for entropy-based sampling · 3.8.5 Further reading for other machine learning models · 3.8.6 Further reading for ensemble-based uncertainty sampling · 3.9 Summary
This introduction motivates diversity sampling through changing model knowledge, rare language, and demographic coverage in a voice assistant.
4.1 Knowing what you don’t know: Identifying gaps in your model’s knowledge · 4.1.1 Example data for diversity sampling · 4.1.2 Interpreting neural models for diversity sampling · 4.1.3 Getting information from hidden layers in PyTorch · 4.2 Model-based outlier sampling · 4.2.1 Use validation data to rank activations · 4.2.2 Which layers should I use to calculate model-based outliers? · 4.2.3 The limitations of model-based outliers · 4.3 Cluster-based sampling · 4.3.1 Cluster members, centroids, and outliers · 4.3.2 Any clustering algorithm in the universe · 4.3.3 K-means clustering with cosine similarity · 4.3.4 Reduced feature dimensions via embeddings or PCA · 4.3.5 Other clustering algorithms · 4.4 Representative sampling · 4.4.1 Representative sampling is rarely used in isolation · 4.4.2 Simple representative sampling · 4.4.3 Adaptive representative sampling · 4.5 Sampling for real-world diversity · 4.5.1 Common problems in training data diversity · 4.5.2 Stratified sampling to ensure diversity of demographics · 4.5.3 Represented and representative: Which matters? · 4.5.4 Per-demographic accuracy · 4.5.5 Limitations of sampling for real-world diversity · 4.6 Diversity sampling with different types of models · 4.6.1 Model-based outliers with different types of models · 4.6.2 Clustering with different types of models · 4.6.3 Representative sampling with different types of models · 4.6.4 Sampling for real-world diversity with different types of models · 4.7 Diversity sampling cheat sheet · 4.8 Further reading · 4.8.1 Further reading for model-based outliers · 4.8.2 Further reading for cluster-based sampling · 4.8.3 Further reading for representative sampling · 4.8.4 Further reading for sampling for real-world diversity · 4.9 Summary
The introduction contrasts model uncertainty with missing knowledge and previews combined sampling and active transfer learning.
5.1 Combining uncertainty sampling and diversity sampling · 5.1.1 Least confidence sampling with cluster-based sampling · 5.1.2 Uncertainty sampling with model-based outliers · 5.1.3 Uncertainty sampling with model-based outliers and clustering · 5.1.4 Representative sampling cluster-based sampling · 5.1.5 Sampling from the highest-entropy cluster · 5.1.6 Other combinations of active learning strategies · 5.1.7 Combining active learning scores · 5.1.8 Expected error reduction sampling · 5.2 Active transfer learning for uncertainty sampling · 5.2.1 Making your model predict its own errors · 5.2.2 Implementing active transfer learning · 5.2.3 Active transfer learning with more layers · 5.2.4 The pros and cons of active transfer learning · 5.3 Applying active transfer learning to representative sampling · 5.3.1 Making your model predict what it doesn’t know · 5.3.2 Active transfer learning for adaptive representative sampling · 5.3.3 The pros and cons of active transfer learning for representative sampling · 5.4 Active transfer learning for adaptive sampling · 5.4.1 Making uncertainty sampling adaptive by predicting uncertainty · 5.4.2 The pros and cons of ATLAS · 5.5 Advanced active learning cheat sheets · 5.6 Further reading for active transfer learning · 5.7 Summary
The introduction uses agricultural object detection to show how active learning choices depend on the task and the kinds of errors that matter.
6.1 Applying active learning to object detection · 6.1.1 Accuracy for object detection: Label confidence and localization · 6.1.2 Uncertainty sampling for label confidence and localization in object detection · 6.1.3 Diversity sampling for label confidence and localization in object detection · 6.1.4 Active transfer learning for object detection · 6.1.5 Setting a low object detection threshold to avoid perpetuating bias · 6.1.6 Creating training data samples for representative sampling that are similar to your predictions · 6.1.7 Sampling for image-level diversity in object detection · 6.1.8 Considering tighter masks when using polygons · 6.2 Applying active learning to semantic segmentation · 6.2.1 Accuracy for semantic segmentation · 6.2.2 Uncertainty sampling for semantic segmentation · 6.2.3 Diversity sampling for semantic segmentation · 6.2.4 Active transfer learning for semantic segmentation · 6.2.5 Sampling for image-level diversity in semantic segmentation · 6.3 Applying active learning to sequence labeling · 6.3.1 Accuracy for sequence labeling · 6.3.2 Uncertainty sampling for sequence labeling · 6.3.3 Diversity sampling for sequence labeling · 6.3.4 Active transfer learning for sequence labeling · 6.3.5 Stratified sampling by confidence and tokens · 6.3.6 Create training data samples for representative sampling that are similar to your predictions · 6.3.7 Full-sequence labeling · 6.3.8 Sampling for document-level diversity in sequence labeling · 6.4 Applying active learning to language generation · 6.4.1 Calculating accuracy for language generation systems · 6.4.2 Uncertainty sampling for language generation · 6.4.3 Diversity sampling for language generation · 6.4.4 Active transfer learning for language generation · 6.5 Applying active learning to other machine learning tasks · 6.5.1 Active learning for information retrieval · 6.5.2 Active learning for video · 6.5.3 Active learning for speech · 6.6 Choosing the right number of items for human review · 6.6.1 Active labeling for fully or partially annotated data · 6.6.2 Combining machine learning with annotation · 6.7 Further reading · 6.8 Summary
This introduction explains how annotation task requirements shape workforce choices, how different workforces can be combined, and why fair working conditions matter.
7.1 Introduction to annotation · 7.1.1 Three principles of good data annotation · 7.1.2 Annotating data and reviewing model predictions · 7.1.3 Annotations from machine learning-assisted humans · 7.2 In-house experts · 7.2.1 Salary for in-house workers · 7.2.2 Security for in-house workers · 7.2.3 Ownership for in-house workers · 7.2.4 Tip: Always run in-house annotation sessions · 7.3 Outsourced workers · 7.3.1 Salary for outsourced workers · 7.3.2 Security for outsourced workers · 7.3.3 Ownership for outsourced workers · 7.3.4 Tip: Talk to your outsourced workers · 7.4 Crowdsourced workers · 7.4.1 Salary for crowdsourced workers · 7.4.2 Security for crowdsourced workers · 7.4.3 Ownership for crowdsourced workers · 7.4.4 Tip: Create a path to secure work and career advancement · 7.5 Other workforces · 7.5.1 End users · 7.5.2 Volunteers · 7.5.3 People playing games · 7.5.4 Model predictions as annotations · 7.6 Estimating the volume of annotation needed · 7.6.1 The orders-of-magnitude equation for number of annotations needed · 7.6.2 Anticipate one to four weeks of annotation training and task refinement · 7.6.3 Use your pilot annotations and accuracy goal to estimate cost · 7.6.4 Combining types of workforces · 7.7 Summary
The introduction explains why annotation quality matters, why agreement is not always straightforward, and why accuracy should be considered against a chance baseline.
8.1 Comparing annotations with ground truth answers · 8.1.1 Annotator agreement with ground truth data · 8.1.2 Which baseline should you use for expected accuracy? · 8.2 Interannotator agreement · 8.2.1 Introduction to interannotator agreement · 8.2.2 Benefits from calculating interannotator agreement · 8.2.3 Dataset-level agreement with Krippendorff’s alpha · 8.2.4 Calculating Krippendorff’s alpha beyond labeling · 8.2.5 Individual annotator agreement · 8.2.6 Per-label and per-demographic agreement · 8.2.7 Extending accuracy with agreement for real-world diversity · 8.3 Aggregating multiple annotations to create training data · 8.3.1 Aggregating annotations when everyone agrees · 8.3.2 The mathematical case for diverse annotators and low agreement · 8.3.3 Aggregating annotations when annotators disagree · 8.3.4 Annotator-reported confidences · 8.3.5 Deciding which labels to trust: Annotation uncertainty · 8.4 Quality control by expert review · 8.4.1 Recruiting and training qualified people · 8.4.2 Training people to become experts · 8.4.3 Machine-learning-assisted experts · 8.5 Multistep workflows and review tasks · 8.6 Further reading · 8.7 Summary
The chapter introduction identifies challenges in subjective annotation, machine-assisted quality control, and imperfect training data.
9.1 Annotation quality for subjective tasks · 9.1.1 Requesting annotator expectations · 9.1.2 Assessing viable labels for subjective tasks · 9.1.3 Trusting an annotator to understand diverse responses · 9.1.4 Bayesian Truth Serum for subjective judgments · 9.1.5 Embedding simple tasks in more complicated ones · 9.2 Machine learning for annotation quality control · 9.2.1 Calculating annotation confidence as an optimization task · 9.2.2 Converging on label confidence when annotators disagree · 9.2.3 Predicting whether a single annotation is correct · 9.2.4 Predicting whether a single annotation is in agreement · 9.2.5 Predicting whether an annotator is a bot · 9.3 Model predictions as annotations · 9.3.1 Trusting annotations from confident model predictions · 9.3.2 Treating model predictions as a single annotator · 9.3.3 Cross-validating to find mislabeled data · 9.4 Embeddings and contextual representations · 9.4.1 Transfer learning from an existing model · 9.4.2 Representations from adjacent easy-to-annotate tasks · 9.4.3 Self-supervision: Using inherent labels in the data · 9.5 Search-based and rule-based systems · 9.5.1 Data filtering with rules · 9.5.2 Training data search · 9.5.3 Masked feature filtering · 9.6 Light supervision on unsupervised models · 9.6.1 Adapting an unsupervised model to a supervised model · 9.6.2 Human-guided exploratory data analysis · 9.7 Synthetic data, data creation, and data augmentation · 9.7.1 Synthetic data · 9.7.2 Data creation · 9.7.3 Data augmentation · 9.8 Incorporating annotation information into machine learning models · 9.8.1 Filtering or weighting items by confidence in their labels · 9.8.2 Including the annotator identity in inputs · 9.8.3 Incorporating uncertainty into the loss function · 9.9 Further reading for advanced annotation · 9.9.1 Further reading for subjective data · 9.9.2 Further reading for machine learning for annotation quality control · 9.9.3 Further reading for embeddings/contextual representations · 9.9.4 Further reading for rule-based systems · 9.9.5 Further reading for incorporating uncertainty in annotations into the downstream models · 9.10 Summary
This introduction illustrates how one application can require several annotation tasks and why annotation quality strategies must be adapted to each task.
10.1 Annotation quality for continuous tasks · 10.1.1 Ground truth for continuous tasks · 10.1.2 Agreement for continuous tasks · 10.1.3 Subjectivity in continuous tasks · 10.1.4 Aggregating continuous judgments to create training data · 10.1.5 Machine learning for aggregating continuous tasks to create training data · 10.2 Annotation quality for object detection · 10.2.1 Ground truth for object detection · 10.2.2 Agreement for object detection · 10.2.3 Dimensionality and accuracy in object detection · 10.2.4 Subjectivity for object detection · 10.2.5 Aggregating object annotations to create training data · 10.2.6 Machine learning for object annotations · 10.3 Annotation quality for semantic segmentation · 10.3.1 Ground truth for semantic segmentation annotation · 10.3.2 Agreement for semantic segmentation · 10.3.3 Subjectivity for semantic segmentation annotations · 10.3.4 Aggregating semantic segmentation to create training data · 10.3.5 Machine learning for aggregating semantic segmentation tasks to create training data · 10.4 Annotation quality for sequence labeling · 10.4.1 Ground truth for sequence labeling · 10.4.2 Ground truth for sequence labeling in truly continuous data · 10.4.3 Agreement for sequence labeling · 10.4.4 Machine learning and transfer learning for sequence labeling · 10.4.5 Rule-based, search-based, and synthetic data for sequence labeling · 10.5 Annotation quality for language generation · 10.5.1 Ground truth for language generation · 10.5.2 Agreement and aggregation for language generation · 10.5.3 Machine learning and transfer learning for language generation · 10.5.4 Synthetic data for language generation · 10.6 Annotation quality for other machine learning tasks · 10.6.1 Annotation for information retrieval · 10.6.2 Annotation for multifield tasks · 10.6.3 Annotation for video · 10.6.4 Annotation for audio data · 10.7 Further reading for annotation quality for different machine learning tasks · 10.7.1 Further reading for computer vision · 10.7.2 Further reading for annotation for natural language processing · 10.7.3 Further reading for annotation for information retrieval · 10.8 Summary
This introduction explains why annotation interfaces should be tailored to the task and workforce to support quality and efficiency.
11.1 Basic principles of human–computer interaction · 11.1.1 Introducing affordance, feedback, and agency · 11.1.2 Designing interfaces for annotation · 11.1.3 Minimizing eye movement and scrolling · 11.1.4 Keyboard shortcuts and input devices · 11.2 Breaking the rules effectively · 11.2.1 Scrolling for batch annotation · 11.2.2 Foot pedals · 11.2.3 Audio inputs · 11.3 Priming in annotation interfaces · 11.3.1 Repetition priming · 11.3.2 Where priming hurts · 11.3.3 Where priming helps · 11.4 Combining human and machine intelligence · 11.4.1 Annotator feedback · 11.4.2 Maximizing objectivity by asking what other people would annotate · 11.4.3 Recasting continuous problems as ranking problems · 11.5 Smart interfaces for maximizing human intelligence · 11.5.1 Smart interfaces for semantic segmentation · 11.5.2 Smart interfaces for object detection · 11.5.3 Smart interfaces for language generation · 11.5.4 Smart interfaces for sequence labeling · 11.6 Machine learning to assist human processes · 11.6.1 The perception of increased efficiency · 11.6.2 Active learning for increased efficiency · 11.6.3 Errors can be better than absence to maximize completeness · 11.6.4 Keep annotation interfaces separate from daily work interfaces · 11.7 Further reading · 11.8 Summary
This chapter introduces three prototype human-in-the-loop machine learning products and their potential extensions.
12.1 Defining products for human-in-the-loop machine learning applications · 12.1.1 Start with the problem you are solving · 12.1.2 Design systems to solve the problem · 12.1.3 Connecting Python and HTML · 12.2 Example 1: Exploratory data analysis for news headlines · 12.2.1 Assumptions · 12.2.2 Design and implementation · 12.2.3 Potential extensions · 12.3 Example 2: Collecting data about food safety events · 12.3.1 Assumptions · 12.3.2 Design and implementation · 12.3.3 Potential extensions · 12.4 Example 3: Identifying bicycles in images · 12.4.1 Assumptions · 12.4.2 Design and implementation · 12.4.3 Potential extensions · 12.5 Further reading for building human-in-the-loop machine learning products · 12.6 Summary
inter-rater reliabilityA measure of annotators’ agreement on labels assigned to tokens, considered separately from span boundaries.
diversity-based samplingA method of selecting unlabeled items that are underrepresented or unknown in the model’s current training data or representation.
data annotationA task in which a category or value is assigned to a data item.
accuracyA measure of an annotator's accuracy, estimated in this context by comparing label distributions across annotators.
active learningActive learning is a machine learning approach in which a learning system selects data for annotation to improve its learning process.
sequence labelingSequence labeling is the task of assigning labels to elements or spans in an ordered sequence, such as a text sequence.
representative samplingA sampling approach that seeks a sample reflecting relevant characteristics of the population.
adaptive transfer learningAn approach that combines active learning with transfer learning to select examples for annotation while adapting knowledge from another task.
uncertainty samplingA method of selecting unlabeled items near a model’s decision boundary, where its predictions are uncertain.
transfer learningTransfer learning is a machine-learning technique that applies knowledge learned from one task or dataset to a different task or dataset.
clusteringA procedure that groups data items according to similarities or patterns in their representations.
Krippendorff's alphaA coefficient that measures agreement among annotators while accounting for expected agreement and allowing for missing annotations and different measurement scales.
Intersection over UnionA measure of overlap between a predicted region and a ground-truth region, calculated as their intersection area divided by their union area.
synthetic dataA generated data example derived from existing data rather than directly collected from a real-world instance.
pretrained modelA pretrained model is a machine-learning model trained on data before being adapted to a particular task.
stratified samplingA sampling method that divides a population into subgroups and samples from each subgroup.
cosine similarityCosine similarity measures the similarity between nonzero vectors using the cosine of the angle between them.
expected agreementThe level of agreement expected from the distribution of labels, rather than from annotators’ matching judgments on individual items.
ground truthReference labels or outputs treated as correct for a data item.
model-based outlier samplingAn outlier-detection approach that uses a model to identify observations that differ substantially from usual data patterns.
outlier detectionThe identification of observations that differ substantially from the usual pattern in a dataset.
transparencyOpenness about the purpose, context, or operation of a project and the information shared with its participants.
model uncertaintyUncertainty about the correctness or reliability of a model's predictions.
K-means clusteringK-means clustering partitions data into a specified number of clusters by iteratively assigning points to their nearest cluster centroid and updating the centroids.
continuous annotation taskAn annotation task in which judgments or labels take values along a continuous numeric scale.
crowdsourced workerA person who completes tasks obtained through a crowd-based labor arrangement.
activationA neural network activation is the output value of a unit or layer during a forward computation.
semi-supervised learningA machine learning approach that learns from a combination of labeled and unlabeled data.
embeddingAn embedding is a learned vector representation of an item in a numerical space.
random samplingRandom selection is a sampling method in which items are chosen at random.
information retrievalThe task of finding and retrieving relevant information from a collection in response to an information need.
uncertainty scoreAn uncertainty score is a numerical measure of how uncertain a model is about a prediction or example.
individual annotator agreementA measure of how consistently one annotator's judgments agree with other annotators or a reference standard.
cluster-based samplingA sampling approach that uses statistical methods independent of a model to select a diverse set of items for labeling.
data augmentationA technique that creates modified or additional data examples to increase the amount or diversity of data available for learning.
affordanceA property of an interface that suggests or enables an action for its user.
validation setA dataset used to assess a model during development, often to guide model selection or tuning.
subjective taskAn annotation task whose judgments can depend on interpretation or perspective.
discounted cumulative gainA ranking metric that scores results by graded relevance while discounting results at lower ranks.
retrainingTraining a machine learning model or part of one again, often using updated data or adapting it to a task.
contextualized representationA representation of an item whose content reflects its surrounding context, such as neighboring words or sentences.
assisted annotationAn annotation interface in which computational assistance helps a person produce or edit annotations.
majority agreementA measure of how often an annotator assigns the label most commonly assigned by other annotators to the same task.
feedbackA response from an interface that indicates whether a user's action was recorded or had an effect.
ratio samplingAn uncertainty-sampling strategy that ranks examples by the ratio between the scores of their two most likely classes.
max margin samplingAn uncertainty-sampling strategy that selects examples with the smallest difference between the scores of the two most likely classes.
random sampleA sample selected randomly from a population of data so that its composition is not deliberately filtered by particular criteria.
localization confidenceThe degree of confidence in a model's predicted spatial location for an object.
predictive annotation interfaceAn annotation interface that presents items pre-annotated by a machine learning model and lets annotators edit them.
adjudication interfaceA user-facing interface for resolving disagreements between annotators or reviewing model predictions for annotation.
object detectionA computer vision task that identifies and locates objects within an image or video.
primingPriming is the influence of context or preceding events on a person's perception.
autocompleteA user-interface feature that suggests or completes text as a person types.
anomaly scoreA numerical score indicating how strongly an item differs from patterns represented in reference data or a model.
logitA logit is an unnormalized score produced by a model before conversion into probabilities.
expert reviewA quality-control method in which a knowledgeable reviewer assesses annotation decisions, especially disputed cases.
contextual word predictionContextual word prediction is a language-modeling task that predicts a word from the surrounding text.
ownershipA worker’s perceived sense of involvement in and responsibility for an annotation project.
entropyA measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome's probability multiplied by its logarithm.
least confident strategyAn active learning strategy that selects examples for which the model's most probable predicted class has the lowest confidence.
job securityJob security is the stability of a person's employment and the likelihood that they can continue in their job.
label confidenceThe degree of confidence a model assigns to its predicted class label for an example.
ensembleA method that combines or compares predictions from multiple models to support a prediction or decision.
baselineA reference result or level of performance against which other results are compared.
margin of errorAn allowed amount of deviation between a measured or predicted value and its reference value.
contextual modelA model trained to predict contextual information or auxiliary labels associated with data, whose learned information can support another task.
bounding boxA rectangular region specified around an object in an image or video frame, commonly used as a visual annotation.
predictive textA text-entry feature that predicts likely upcoming words or text as a person types.
in-house personnelA person who performs work within the organization that needs it.
expected error reduction samplingAn active learning strategy that selects items expected to produce the greatest reduction in model error when labeled.
review taskAn annotation task in which annotators review candidate labels or sequences rather than label raw data from scratch.
intersectional biasBias that varies across combinations of demographic characteristics rather than affecting each characteristic independently.
dimensionality reductionDimensionality reduction is the process of representing data with fewer dimensions while retaining relevant information.
principal component analysisPrincipal component analysis is a statistical technique that transforms correlated features into a smaller set of uncorrelated components.
pairwise rankingA ranking method that establishes relative order by asking annotators to compare pairs of items.
posteditingThe editing of machine-generated translation output by a human.
Bayesian Truth SerumA scoring method that combines respondents’ actual answers with their predictions about the frequency of possible answers to identify answers that are more common than predicted.
real-world diversityThe variation in characteristics and experiences across people or groups in the population relevant to a data or machine learning task.
observed agreementThe agreement actually observed among annotators' responses to the same items.
expert annotatorAn annotator with expertise relevant to the annotation task.
repetition primingRepetition priming is a form of priming in which the sequence of tasks influences a person's perception.
IOB2 taggingA sequence-labeling scheme that uses outside, beginning, and inside tags to mark labeled spans and their boundaries.
light supervisionA form of human guidance that influences a learning or analysis process without requiring full direct labeling of every item.
fair compensationRemuneration that is adequate and equitable in relation to workers’ circumstances and needs.
keyboard shortcutA key or key combination that triggers an action without requiring menu navigation or pointer input.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.