liveBook · Manning

Data acquisition strategy

As explained in LLM Customization and Fine-Tuning

The plan for collecting, selecting, and preparing data to support model adaptation.

A large dataset is not necessarily ready for model adaptation: operational records may have inconsistent categories, missing labels, or answers scattered across systems. Choose collection and labeling methods around data quality, available resources, and the production distribution. section 3.0

Four strategies trade off human effort, cost, label quality, and freshness; production datasets often combine them.

  • Human annotation provides high-quality seed data, evaluation sets, and labels for high-stakes edge cases. section 3.4
  • Weak supervision applies programmable labeling rules to high-volume categories with clear criteria. section 3.9
  • Synthetic generation fills rare-category gaps and adds controlled variation; use a quality gate. section 3.4
  • Production log mining provides fresh examples aligned with changing production distributions, but label quality varies. section 3.4
Table 3.2 Different data acquisition strategies and what is best used for
Strategy
Relative cost
Label quality
Best used for
Human annotation
High
Highest
Seed data, evaluation sets, high-stakes edge cases
Weak supervision
Low
Moderate
High-volume categories with clear labeling rules
Synthetic generation
Medium
High (with gate)
Rare categories, gap-filling, controlled variation
Production log mining
Low
Variable
Distribution alignment, temporal freshness

Compare the four strategies by relative cost, label quality, and best use. section 3.4

Combining methods for IT support

For an IT-support assistant, anchor quality with a small human-labeled seed, use weak supervision for common categories, generate examples for rare categories, and add rolling production samples. Add synthetic data where evaluation finds gaps, keeping it below 30% of the dataset. section 3.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Compares four methods by cost, label quality, and use, then gives combination guidance rather than a universal mix.
Examples
Uses an IT-support assistant combining labeled tickets, weak supervision, synthetic rare-category examples, and production samples.
What you can do after
Apply a mix of acquisition methods to balance budget, timeline, and quality targets.
How it connects
Links human annotation, weak supervision, synthetic data generation, production log mining, and model evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.