A training dataset is the collection of examples used to fit or adapt a model. Its quality and coverage determine what the model can learn: inconsistent labels, missing cases, and unrepresentative records can produce confidently wrong behavior that is hard to diagnose after deployment. section 3.0
Enterprise records are rarely ready to train on. Support tickets, chats, and emails may contain drifting categories, incomplete fields, signatures, timestamps, and personal information. Curation, filtering, privacy scrubbing, and normalization come before conversion into a training format. section 3.0 section 6.7
Build the dataset around the behavior the model must produce:
- Select representative, diverse examples. Volume alone is not enough; noisy or inconsistent responses teach the model those same inconsistencies. section 6.7
- Match volume to task complexity. Simple classification and extraction generally need fewer examples than domain-expert reasoning; treat published ranges as starting points for experiments, not guarantees. section 6.7
- Use synthetic examples to supplement human-labeled data, not to contaminate evaluation. One adaptation workflow starts with about 50 labeled seed examples and caps synthetic data at 30% of the training set. section 3.7.5
- Keep evaluation data separate. Freeze a golden set with expert reference answers, and exclude it from training; use separate validation and test sets where appropriate. section 3.7.5 section 6.12.1
Table 6.8 Data volume guidance for full SFT by task complexity
|
Task complexity
|
Minimum viable dataset
|
Recommended dataset
|
Diminishing returns threshold
|
|
Simple (classification, extraction)
|
100 to 200 examples
|
300 to 500 examples
|
~1,000 examples
|
|
Moderate (summarization, Q&A, structured
output)
|
300 to 500 examples
|
500 to 1,500 examples
|
~3K examples
|
|
Complex (multi-step reasoning, novel format)
|
500 to 1,000 examples
|
1,500 to 5,000 examples
|
~10K examples
|
|
Highly complex (domain-expert-level reasoning)
|
1,000 to 2,000 examples
|
3,000 to 10,000 examples
|
~20K+ examples
|
Compare starting data-volume ranges for full supervised fine-tuning by task complexity, including approximate diminishing-returns thresholds. section 6.7
For supervised fine-tuning, examples can be formatted as chat-structured JSONL conversations with system, user, and assistant turns. Loss is computed on assistant tokens; system and user turns provide context but are masked from the loss. Assistant responses therefore need to reflect the desired terminology and format. section 6.7
Preparing clinical fine-tuning data
A clinical assistant project selected 5,200 examples from a medical reasoning collection across general knowledge, clinical reasoning, drug interactions, and patient communication. Each example paired a clinical question with structured reasoning and a concise answer. Deduplication, length filtering, and review of a stratified 200-example sample removed about 8% of the initial data. The team held out 520 examples for validation and assembled a separate 200-example test set from a different time period to guard against temporal leakage. section 6.12.1
Teaching an internal support style
An IT support assistant used handwritten question-and-answer seeds expanded to 166 examples with a teacher model and filtered to a house format. Six examples were held out. On those questions, a LoRA adapter trained on the set named internal tools and followed the required format, while the base and format-prompted models did neither. section 5.1.8
Before choosing a more advanced optimization technique, inspect the training data for noise, inconsistency, and gaps in the target distribution. Improving those examples can be a better investment than applying preference optimization to mediocre data. section 8.8.1

















































