Suppose a team wants an AI system to sort client documents into the categories its clients use. A collection of documents is not enough: each training example needs a label identifying the intended category. Those labels tell the model what result to learn, and they must reflect the categories users actually need. section 8.2.2 section 3.1
Labeled data pairs an input with a target result. A review paired with its sentiment or an image paired with its animal is labeled; the same input without that extra signal is unlabeled. Labels enable supervised learning, including classification. They also constrain the task: a classifier trained on user behavior and existing segment labels can learn to predict segments for new users. section 3.1.2 section 4.2.2
Choose how to obtain labels with their limitations in mind:
- Reuse existing annotations or proxy labels, such as star ratings for review sentiment. Check whether that data differs from real users’ data or reflects self-selection bias. section 4.2.1
- Use an LLM to label simpler tasks, but plan for fine-tuning or substantial human oversight when the task is complex or domain-specific. section 4.2.1
- Collect labels during product use, such as ratings of recommendations. Make giving feedback worthwhile enough to obtain sufficient data. section 4.2.1
- Organize human annotation with clear instructions and suitable tools. Account for fatigue and subjective bias, then review sampled labels for consistency. section 4.2.1 section 8.2.2
Table 8.2 Data sample for supervised fine-tuning
| Input |
Target output |
|
| Document title |
Text excerpt |
Label |
| “API integration guide” |
“Detailed steps to integrate workflows . . .” |
Workflow automation |
| “Compliance checklist 2024” |
“Ensures adherence to data protection . . .” |
Compliance and security |
| “Top-10 trends in IT governance” |
“In this article, we will present the top trends in compliance . . .” |
Compliance and security |
| “AI trends in automation” |
“Examining the latest advances in ML . . .” |
Cutting-edge automation |
The document examples pair titles and excerpts with target categories. Before supervised fine-tuning, check that the categories match the client’s taxonomy, add examples for underrepresented categories, and sample records to verify their labels. section 8.2.2
More labels are not automatically better. An unbalanced dataset can favor common categories; inconsistent labels can weaken the learning signal. Dataset preparation is therefore a product decision as well as a training task: agree on the desired outputs, inspect where examples come from, and review whether the labeled cases represent the work the system will face. section 8.2.2 section 4.2.1
