liveBook · Manning

Data annotation

As explained in LLM Customization and Fine-Tuning

The process of adding labels or other explanatory information to data so it can support model development or evaluation.

Human annotation pairs data with labels assigned by people who understand the task. It produces high-quality, task-specific ground truth, but takes time and costs more than automated approaches, so reserve it for cases where label quality matters most. section 3.4.1

Use annotation deliberately:

  • Create a clear labeling brief: define labels, specify rules for recurring boundary cases, and give three to five worked examples per label. section 3.4.1
  • Use human-validated labels for much of the training data when appropriate, but keep the golden evaluation set fully human-annotated and held out. section 3.4.1
  • Focus annotation on seed data, evaluation sets, and high-stakes edge cases; combine it with other acquisition strategies when scale or speed matters. section 3.4

Ticket triage

A support engineer labels “Can't print to the 3rd floor printer since this morning, tried restarting it twice” as hardware, severity P3. For a separate ambiguous case, the brief’s boundary rules help annotators apply labels consistently. section 3.4 section 3.4.1

Table 3.2 Different data acquisition strategies and what is best used for
Strategy
Relative cost
Label quality
Best used for
Human annotation
High
Highest
Seed data, evaluation sets, high-stakes edge cases
Weak supervision
Low
Moderate
High-volume categories with clear labeling rules
Synthetic generation
Medium
High (with gate)
Rare categories, gap-filling, controlled variation
Production log mining
Low
Variable
Distribution alignment, temporal freshness

Compare annotation with other data acquisition strategies by relative cost, label quality, and best use. section 3.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines label sets, boundary rules, worked examples, and annotation agreement, emphasizing the labeling brief’s role in consistent judgments.
Examples
Uses IT ticket triage with network, access, hardware, and software categories, P1–P4 severity labels, and a printer-ticket example.
What it assumes
Assumes task-specific label definitions and knowledgeable annotators; specifies how to document labels and resolve recurring boundary cases.
What you can do after
Evaluate model performance against a held-out golden set whose labels are fully human-annotated.
How it connects
Connects human annotation with weak supervision, synthetic data, production log mining, ground truth, and evaluation sets.

Also explained in

Compare how 14 books explain data annotation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.