liveBook · Manning

Training dataset

As explained in LLM Customization and Fine-Tuning

The collection of examples used to fit or adapt a model during training.

A training dataset is the collection of examples used to fit or adapt a model. Its quality and coverage determine what the model can learn: inconsistent labels, missing cases, and unrepresentative records can produce confidently wrong behavior that is hard to diagnose after deployment. section 3.0

Enterprise records are rarely ready to train on. Support tickets, chats, and emails may contain drifting categories, incomplete fields, signatures, timestamps, and personal information. Curation, filtering, privacy scrubbing, and normalization come before conversion into a training format. section 3.0 section 6.7

Build the dataset around the behavior the model must produce:

  • Select representative, diverse examples. Volume alone is not enough; noisy or inconsistent responses teach the model those same inconsistencies. section 6.7
  • Match volume to task complexity. Simple classification and extraction generally need fewer examples than domain-expert reasoning; treat published ranges as starting points for experiments, not guarantees. section 6.7
  • Use synthetic examples to supplement human-labeled data, not to contaminate evaluation. One adaptation workflow starts with about 50 labeled seed examples and caps synthetic data at 30% of the training set. section 3.7.5
  • Keep evaluation data separate. Freeze a golden set with expert reference answers, and exclude it from training; use separate validation and test sets where appropriate. section 3.7.5 section 6.12.1
Table 6.8 Data volume guidance for full SFT by task complexity
Task complexity
Minimum viable dataset
Recommended dataset
Diminishing returns threshold
Simple (classification, extraction)
100 to 200 examples
300 to 500 examples
~1,000 examples
Moderate (summarization, Q&A, structured output)
300 to 500 examples
500 to 1,500 examples
~3K examples
Complex (multi-step reasoning, novel format)
500 to 1,000 examples
1,500 to 5,000 examples
~10K examples
Highly complex (domain-expert-level reasoning)
1,000 to 2,000 examples
3,000 to 10,000 examples
~20K+ examples

Compare starting data-volume ranges for full supervised fine-tuning by task complexity, including approximate diminishing-returns thresholds. section 6.7

For supervised fine-tuning, examples can be formatted as chat-structured JSONL conversations with system, user, and assistant turns. Loss is computed on assistant tokens; system and user turns provide context but are masked from the loss. Assistant responses therefore need to reflect the desired terminology and format. section 6.7

Preparing clinical fine-tuning data

A clinical assistant project selected 5,200 examples from a medical reasoning collection across general knowledge, clinical reasoning, drug interactions, and patient communication. Each example paired a clinical question with structured reasoning and a concise answer. Deduplication, length filtering, and review of a stratified 200-example sample removed about 8% of the initial data. The team held out 520 examples for validation and assembled a separate 200-example test set from a different time period to guard against temporal leakage. section 6.12.1

Teaching an internal support style

An IT support assistant used handwritten question-and-answer seeds expanded to 166 examples with a teacher model and filtered to a house format. Six examples were held out. On those questions, a LoRA adapter trained on the set named internal tools and followed the required format, while the base and format-prompted models did neither. section 5.1.8

Before choosing a more advanced optimization technique, inspect the training data for noise, inconsistency, and gaps in the target distribution. Improving those examples can be a better investment than applying preference optimization to mediocre data. section 8.8.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Details curation, volume guidance, chat formatting, loss masking, and evaluation separation, while deferring multi-turn packing and turn-level masking.
Examples
Uses clinical reasoning examples and synthetic-expanded IT support question-and-answer pairs to illustrate preparation and adaptation.
What you can do after
Evaluate and prepare adaptation data by checking coverage, quality, formatting, and separation from evaluation sets.
How it connects
Connects training datasets to supervised fine-tuning, LoRA, synthetic data, loss masking, and evaluation.

Also explained in

Compare how 50 books explain training dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.