Human annotation pairs data with labels assigned by people who understand the task. It produces high-quality, task-specific ground truth, but takes time and costs more than automated approaches, so reserve it for cases where label quality matters most. section 3.4.1
Use annotation deliberately:
- Create a clear labeling brief: define labels, specify rules for recurring boundary cases, and give three to five worked examples per label. section 3.4.1
- Use human-validated labels for much of the training data when appropriate, but keep the golden evaluation set fully human-annotated and held out. section 3.4.1
- Focus annotation on seed data, evaluation sets, and high-stakes edge cases; combine it with other acquisition strategies when scale or speed matters. section 3.4
Ticket triage
A support engineer labels “Can't print to the 3rd floor printer since this morning, tried restarting it twice” as hardware, severity P3. For a separate ambiguous case, the brief’s boundary rules help annotators apply labels consistently. section 3.4 section 3.4.1
Table 3.2 Different data acquisition strategies and what is best used for
|
Strategy
|
Relative cost
|
Label quality
|
Best used for
|
|
Human
annotation
|
High
|
Highest
|
Seed data,
evaluation sets, high-stakes edge cases
|
|
Weak
supervision
|
Low
|
Moderate
|
High-volume
categories with clear labeling rules
|
|
Synthetic
generation
|
Medium
|
High (with
gate)
|
Rare
categories, gap-filling, controlled variation
|
|
Production log
mining
|
Low
|
Variable
|
Distribution
alignment, temporal freshness
|
Compare annotation with other data acquisition strategies by relative cost, label quality, and best use. section 3.4













