Data annotation
Data annotation is the process of assigning labels, metadata, or structured judgments to data according to specified criteria.
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
A task in which a category or value is assigned to a data item.
- How it is used
- Creates labeled data for training, validation, and evaluation within a human-in-the-loop machine learning workflow.
- How deep it goes
- Develops annotation from simple labeling to workforce planning, interface tradeoffs, quality control, and model-assisted review.
- Examples
- Uses sentiment forms, video bounding boxes, pedestrian detection, and search-result feedback to illustrate annotation choices and risks.
- What you can do after
- Apply annotation planning to choose a workforce, refine guidelines, and weigh model-assisted review against quality and sampling risks.
Machine Learning System Design
The process of assigning annotations or target values to data examples for machine learning.
- How it is used
- Defines target labels for machine learning datasets in line with product goals and available label sources.
- How deep it goes
- Outlines label-format tradeoffs and an operational checklist while leaving platform and staffing choices open in a design example.
- Examples
- Uses malignancy labels for medical images and relevance labels for query–image pairs.
- What you can do after
- Evaluate annotation choices against product needs, labeling capacity, and cost.
Read how Machine Learning System Design explains data annotation →
LLMs in Production
Data annotation is the process of attaching labels or metadata to data instances, often with awareness of their positions or spans.
- How it is used
- Distinguishes annotation from the data cleaning still needed for self-supervised language model training.
- How deep it goes
- Mentions missing annotations in text datasets but does not explain how to create them.
- Examples
- Uses large parquet or text files without labels or annotations as examples of training data.
- What you can do after
- Evaluate whether missing annotations necessarily prevent self-supervised language model training.
Deep Learning with R, Third Edition
Data annotation is the process of assigning human-provided labels or other metadata to data examples.
- How it is used
- Supplies targets for supervised learning and guides decisions about collecting, labeling, and preparing training data.
- How deep it goes
- Explains annotation purpose, labor, quality effects, sourcing options, expertise requirements, and noise tradeoffs without detailing labeling procedures.
- Examples
- Uses sentiment-labeled movie reviews, cat-versus-dog images, dog breeds, and CT scans of bone fractures.
- What you can do after
- Evaluate annotation needs and choose labelers according to task complexity, expertise, cost, scale, control, and noise.
Read how Deep Learning with R, Third Edition explains data annotation →
Deep Learning with Python, Third Edition
The process of having people manually label or correct dataset examples.
- How it is used
- Connects supervised-learning inputs to target labels and treats annotation as a core dataset-collection task.
- How deep it goes
- Extends from manual labels and provider tradeoffs to expert correction in a semiautomated segmentation pipeline.
- Examples
- Uses cat-versus-dog labels, dog-breed classification, fracture scans, and SA-1B object masks as annotation examples.
- What you can do after
- Apply it by selecting suitable labelers and weighing expertise, cost, scalability, control, and label noise.
Read how Deep Learning with Python, Third Edition explains data annotation →
Effective Conversational AI
Data annotation is the process of assigning labels or judgments to data examples for training, testing, or evaluation.
- How it is used
- Measures and improves a classifier’s understanding of real user utterances for training, testing, or evaluation.
- How deep it goes
- Explains filtering, labeling, collaboration, disagreement resolution, and prediction review, but stops short of implementation details.
- Examples
- Uses customer-service utterances labeled Request_Agent or Office_Hours and pairs flight-baggage or refund questions with expected responses.
- What you can do after
- Diagnose/debug with it by producing golden labels, resolving ambiguous intents, and constructing representative test data.
Read how Effective Conversational AI explains data annotation →
Machine Learning Platform Engineering
An annotation is metadata attached to an input example that identifies or describes a target property for machine learning.
- How it is used
- Uses annotations to prepare labeled examples for training and evaluating machine-learning models.
- How deep it goes
- Explains modality-specific labeling, dataset splitting, and data-versioning implications without detailing annotator quality controls or tool internals.
- Examples
- Demonstrates OCR bounding boxes, handwritten ID-number transcription, fraud labels, and annotated identity cards in CVAT.
- What you can do after
- Apply annotation through CVAT and organize labeled examples for model training, validation, and testing.
Read how Machine Learning Platform Engineering explains data annotation →
Outlier Detection in Python
Labeling is the process of assigning annotations or target values to data records.
- How it is used
- Uses human judgments to classify flagged records, guide investigation, and improve future outlier detection.
- How deep it goes
- Explains binary, categorical, and scored labels plus downstream uses, but stops short of a detailed annotation workflow.
- Examples
- Uses financial transactions, unusual staff, sales combinations, and rare times of day as annotation examples.
- What you can do after
- Apply it to categorize flagged records, prioritize investigation, retune detectors, and train downstream rules or models.
Read how Outlier Detection in Python explains data annotation →
Designing Deep Learning Systems: A guide for software engineers
The process of assigning annotations or target labels to data examples for supervised machine learning.
- How it is used
- Supports data exploration by turning collected examples into datasets for later model research and prototyping.
- How deep it goes
- Places labeling in an informal early workflow and stops short of specifying annotation standards, quality controls, or a formal collection pipeline.
- Examples
- Uses public or internal data, outsourced or in-house labeling, Python or shell scripts, manual copying, and web-based analysis tools.
- What you can do after
- Build it from scratch by collecting, labeling, and assembling datasets for algorithm evaluation and proof-of-concept models.
Read how Designing Deep Learning Systems: A guide for software engineers explains data annotation →
The Art of AI Product Development
Data annotation is the process of adding labels or other structured annotations to raw data.
- How it is used
- Supports supervised learning by converting raw data into learning signals for classification, prediction, and few-shot prompting.
- How deep it goes
- Explains label sources, quality checks, human risks, and automation, but stops short of detailed annotation-tool implementation.
- Examples
- Uses sentiment labels, animal-image labels, next-word pairs, user-segment tables, and retrieved question-answering examples.
- What you can do after
- Apply it by designing labeling guidelines, checking agreement and class balance, and selecting human or automated annotation.
Read how The Art of AI Product Development explains data annotation →
LLM Customization and Fine-Tuning
The process of adding labels or other explanatory information to data so it can support model development or evaluation.
- How it is used
- Anchors model development with high-quality labels, especially for seed data, evaluation sets, and high-stakes edge cases.
- How deep it goes
- Defines label sets, boundary rules, worked examples, and annotation agreement, emphasizing the labeling brief’s role in consistent judgments.
- Examples
- Uses IT ticket triage with network, access, hardware, and software categories, P1–P4 severity labels, and a printer-ticket example.
- What you can do after
- Evaluate model performance against a held-out golden set whose labels are fully human-annotated.
Read how LLM Customization and Fine-Tuning explains data annotation →
Building Specialized AI Systems
Labeling is the process of pairing input data with desired outputs or annotations to create examples for model training.
- How it is used
- Turns prepared datasets into supervised fine-tuning examples by specifying the response a model should learn.
- How deep it goes
- Explains pairing, consistency, output design, and downstream effects, but stops short of detailed annotation operations.
- Examples
- Uses suspicious-versus-legitimate transactions, utility categories, and client-facing transaction summaries.
- What you can do after
- Build training examples by defining consistent labels and selecting output formats aligned with the intended model behavior.
Read how Building Specialized AI Systems explains data annotation →
Hidden Influences
The process of assigning descriptive or evaluative labels to content so that it can be identified, classified, or analyzed.
- How it is used
- Uses labels to support content-based recommendations and to identify content for measuring recommender amplification.
- How deep it goes
- Explains labeling through examples and discusses cost, consistency, scale, nuance, and privacy concerns.
- Examples
- Uses book genres and regions, plus social posts supporting or opposing a policy and users’ political stances.
- What you can do after
- Evaluate how labeling choices and constraints affect recommendation quality and amplification measurement.
Building Reliable AI Systems
The process of adding human-provided labels, rankings, or other judgments to data.
- How it is used
- Supplies human labels or judgments when fine-tuning data needs document classes or ranked response quality.
- How deep it goes
- Explains when manual labels and ratings are needed, but stops short of a labeling protocol or reviewer-agreement method.
- Examples
- Uses legal-document issue tags and human ratings or annotations of response politeness and thoroughness as examples.
- What you can do after
- Practice/perform it by tagging domain documents or judging outputs against a quality goal, while favoring accurate, domain-reviewed examples.
Read how Building Reliable AI Systems explains data annotation →
Also covered in
- Fundamentals of CybersecurityIn early access (MEAP)
- Building LLM Applications with DSPy
- AI-Powered Search
- Software Engineering for Data ScientistsIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- machine learningMachine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
Related concepts
- labelproduces · A label is a target value, category, or reference annotation associated with an example for learning, prediction, or evaluation.
- labeled dataproduces · Labeled data is a collection of examples paired with target values, categories, or other task-relevant annotations.
- training datasetproduces · A training dataset is a collection of examples supplied to a learning procedure to fit or adapt a model’s parameters or behavior.
- content-based recommenderused by · A content-based recommender suggests items whose features resemble those of items matching a user’s interests or prior interactions.
- active learningused by · Active learning is a machine-learning approach in which a learner selects informative data examples for labeling or feedback to improve a model.
- AI modelused by · An AI model is a computational system that learns or encodes patterns to map inputs to outputs such as predictions, classifications, or generated content.
Questions about data annotation
What is data annotation?
Data annotation is the process of assigning labels, metadata, or structured judgments to data according to specified criteria.
Which Manning books explain data annotation?
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, Machine Learning System Design, LLMs in Production, Deep Learning with R, Third Edition, Deep Learning with Python, Third Edition, Effective Conversational AI, Machine Learning Platform Engineering, Outlier Detection in Python and 6 more.
What should I know before learning data annotation?
Manning books that teach data annotation usually introduce language model, accuracy, fine-tuning and machine learning first.
What is data annotation also called?
Data annotation is also called data labeling, human annotation, manual labeling or human labeling.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.













