liveBook · Manning

Data quality

As explained in LLM Customization and Fine-Tuning

The degree to which adaptation data is correct, relevant, and representative enough to support reliable model behavior.

Years of support tickets can look like a ready-made training set until categories drift, severity fields are empty, and automated closures masquerade as resolutions. Data quality determines whether those records can support reliable model adaptation, not just whether they are available in sufficient volume. section 3.0

Adaptation data should contain correct, complete task-level responses, reflect the intended production distribution, and retain traceable origins. Models can learn errors and inconsistent conventions along with useful patterns; a larger noisy dataset is not necessarily better than a smaller curated one. section 3.9 section 6.7.1

Use a repeatable quality checklist before training and whenever the dataset changes:

  • Review responses for factual correctness, completeness, and consistent formatting. Include the context needed to answer; when information is missing, demonstrate clarification rather than guessing. section 5.1.7 section 6.7.1
  • Cover varied inputs, users, scenarios, and difficulty levels. Match expected response types and difficulty where practical, but deliberately overrepresent rare, high-cost cases such as safety refusals. section 5.1.7 section 6.7.1
  • Measure agreement among annotators and improve labeling guidelines when judgments differ. Keep evaluation data separate from training and test regularly on unseen, current queries. section 5.1.7 section 5.10
  • Scrub personal data before examples reach annotators, annotation platforms, or training infrastructure. Review indirect identifiers as well as structured identifiers; removing data later does not remove it from a trained model. section 3.9

Choose acquisition methods for their strengths: human annotation is valuable for seed and evaluation data, weak supervision scales labeling, and production logs provide realistic, fresh inputs. Synthetic examples can fill rare categories, but need a quality gate and distribution checks; a maximum 30% synthetic mix is a heuristic, not a guarantee. section 3.9

IT support fine-tuning

A curated support dataset combines practitioner-written Stack Exchange questions and accepted answers with a small general-knowledge slice. Answers follow a one-line summary and numbered steps, source URLs are recorded, and length filters remove truncated or excessively long examples. A 450-example training subset supports fast iteration, while production use calls for broader checks including agreement, bias, and privacy. section 5.1.1

Filtering clinical reasoning

In a healthcare adaptation, quality filtering removed 8% of examples that would have introduced inconsistent reasoning patterns. Early unfiltered experiments sometimes produced confident but internally contradictory reasoning, illustrating why final-answer correctness alone is insufficient. section 6.12.6

How LLM Customization and Fine-Tuning covers it

How deep it goes
Provides a curation checklist and acquisition tradeoffs, including privacy, distribution, annotation agreement, and synthetic-data quality gates.
Examples
Uses a curated IT support dataset and a healthcare case with 8% of examples removed during filtering.
What you can do after
Evaluate and improve adaptation datasets using review, distribution checks, privacy safeguards, and held-out evaluation.
How it connects
Connects data quality to supervised fine-tuning, LoRA, synthetic data, inter-annotator agreement, and safety evaluation.

Also explained in

Compare how 15 books explain data quality →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.