A large dataset is not necessarily ready for model adaptation: operational records may have inconsistent categories, missing labels, or answers scattered across systems. Choose collection and labeling methods around data quality, available resources, and the production distribution. section 3.0
Four strategies trade off human effort, cost, label quality, and freshness; production datasets often combine them.
- Human annotation provides high-quality seed data, evaluation sets, and labels for high-stakes edge cases. section 3.4
- Weak supervision applies programmable labeling rules to high-volume categories with clear criteria. section 3.9
- Synthetic generation fills rare-category gaps and adds controlled variation; use a quality gate. section 3.4
- Production log mining provides fresh examples aligned with changing production distributions, but label quality varies. section 3.4
Table 3.2 Different data acquisition strategies and what is best used for
|
Strategy
|
Relative cost
|
Label quality
|
Best used for
|
|
Human
annotation
|
High
|
Highest
|
Seed data,
evaluation sets, high-stakes edge cases
|
|
Weak
supervision
|
Low
|
Moderate
|
High-volume
categories with clear labeling rules
|
|
Synthetic
generation
|
Medium
|
High (with
gate)
|
Rare
categories, gap-filling, controlled variation
|
|
Production log
mining
|
Low
|
Variable
|
Distribution
alignment, temporal freshness
|
Compare the four strategies by relative cost, label quality, and best use. section 3.4
Combining methods for IT support
For an IT-support assistant, anchor quality with a small human-labeled seed, use weak supervision for common categories, generate examples for rare categories, and add rolling production samples. Add synthetic data where evaluation finds gaps, keeping it below 30% of the dataset. section 3.4
