liveBook · Manning

Chapter 3: What data do I need for model adaptation?

What you will learn in chapter 3 of LLM Customization and Fine-Tuning

This chapter introduction explains why having enterprise data is not the same as having data suitable for model adaptation, using IT support records to show why quality and representativeness matter.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduction explains why having enterprise data is not the same as having data suitable for model adaptation, using IT support records to show why quality and representativeness matter.

Key terms: AI-ready data, high-quality, representative data, data quality, ticketing system, resolved threads, knowledge base, support chat logs

3.1 Data readiness spectrum

This section explains how to judge whether an adaptation dataset is ready, why to start with human-written examples, and what to check before expanding the data.

Key terms: data readiness spectrum, Level 1, Level 2, clean example, base, human seeds, synthetic generation, base model, labeled examples, data quality

3.2 Why data quality is important

This section explains how flawed or untrusted adaptation data can damage model performance, fairness, reliability, and security.

Key terms: adaptation data, imbalanced labels, swapped fields, truncated values, precision, trust and reliability, bias and fairness, accuracy, training efficiency and cost, poisoning

3.3 How much data do you need?

The section explains why a few hundred high-quality, representative examples can support an initial fine-tuning test, while later improvement may require substantially more data.

Key terms: first useful run, quality ceiling, edge cases, 80/20 split, quality before quantity, representative data, evaluate early

3.3.1 Base models versus instruct models

This section explains how the starting model affects training-data needs and why training examples must match the prompts used at inference.

Key terms: base model, instruct model, instruction-following, instruction, user turn, assistant turn, inference time, chat template

3.3.2 How accuracy responds to data volume

This section explains why fine-tuning accuracy gains diminish as training data grows and why the first examples deserve careful selection.

Key terms: accuracy, training examples, plateau, marginal gain, clean, representative, diverse

3.3.3 Does your training data reflect your production distribution?

This section shows how to compare training tickets with the production queue and prioritize gaps in important scenarios.

Key terms: production sample, training set, scenario categories, distributions, resolved-ticket exports, P1 outages, high-risk, overall example counts

3.3.4 Edge cases matter disproportionately

This section explains why fine-tuning datasets need deliberately sourced hard cases, boundary conditions, and error-handling examples rather than simply mirroring common production queries.

Key terms: example distribution, production queries, edge cases, business risk, hardest variant, boundary conditions, distinct scenarios, target dataset, error handling, incident thread

3.3.5 Diversity across user types, conditions, and variations

Training data must reflect the users and conditions a model will encounter, not just the topics of their queries.

Key terms: representativeness, user population, response depth, systematic biases, seniority level, functional role, market context, sufficient representation, market conditions, temporal variation

3.3.6 Diminishing returns: Why the first 100 examples do most of the work

This section explains how diminishing returns should shape fine-tuning data collection and evaluation.

Key terms: high-quality, representative examples, accuracy gain, golden seed, golden test set, hold out, failure cases, missing coverage, held-out golden set, saturated the current distribution

3.4 Data acquisition strategies

This section explains how to combine data acquisition strategies and choose a starting approach based on available data, effort, and quality needs.

Key terms: human annotation, weak supervision, synthetic generation, production log mining, cost-quality-speed space, seed dataset, quality gate, well-labeled examples, hard variants, synthetic proportion

3.4.1 Human annotation

This section explains when to use human annotation, how a clear labeling brief improves agreement, and why the golden evaluation set needs independent human labels.

Key terms: human annotation, ground truth, severity definitions, labeling brief, label set, disambiguation rules, worked examples, Cohen’s kappa, human-validated, fully human-annotated

3.4.2 Weak supervision

This section explains how weak supervision uses noisy labeling rules and majority voting to label large ticket corpora, then uses abstentions and human review to improve the results.

Key terms: weak supervision, unlabeled corpus, labeling rules, labeling functions, majority vote, non-abstaining functions, abstention rate, spot check, weak coverage, LLM-as-judge

3.4.3 Synthetic data from teacher models

This section explains how to expand a small real dataset with teacher-generated examples while preserving its style, checking examples against the task, and keeping real data dominant.

Key terms: teacher model, seed dataset, anchoring, distributional shift, quality gate, hallucination, classification, summarization, synthetic share, 30% cap

3.4.4 Production log mining

This section explains why production logs are a representative source of training examples and outlines a four-step process for mining them safely and usefully.

Key terms: production log mining, production environment, query and response pairs, PII scrubbing, deduplication, normalized query text, stratified sampling, category distribution

3.5 Data versioning and experiment tracking

The section explains how versioned data, training-run records, and example-level provenance make fine-tuning reproducible and easier to diagnose.

Key terms: data versioning, experiment tracking, regressions, regulatory audit trail, content-hash, manifest, dataset SHA-256, hyperparameter configuration, per-category evaluation scores, model checkpoint

3.5.1 Experiment logging

This subsection defines what an LLM fine-tuning experiment record must capture so runs can be reproduced and compared.

Key terms: experiment logging, experiment record, data, configuration, infrastructure, results

3.5.2 Choosing a tracking tool

This section explains what W&B and MLflow track, how to enable their logging in SFTTrainer, and how data policy affects the choice between them.

Key terms: hyperparameters, training metrics, dataset versioning, SFTTrainer, SFTConfig, report_to, W&B, MLflow, training metadata, data governance

3.5.3 Data lineage

This section explains how per-example data lineage and dataset versioning make training data auditable and help diagnose model regressions.

Key terms: lineage, source provenance, annotation provenance, task brief version, agreement scores, transformation provenance, PII scrubbing, audit trail, dataset manifest, lineage chain

3.6 Generating synthetic training data

Introduces synthetic training data generation as using a stronger teacher LLM to create examples that are verified before joining real seed data.

Key terms: synthetic generation, teacher, labeled examples, seed data

3.6.1 Step 1: Load seed examples from Hugging Face

This subsection chooses public, high-agreement Financial PhraseBank examples and groups them by sentiment to serve as style anchors for synthetic-data generation.

Key terms: Financial PhraseBank, public, licensed dataset, domain-agnostic, AllAgree split, style anchors, sentiment category, category-matched seeds, style anchoring, minority classes

3.6.2 Step 2: Build the prompt with style anchoring

This section explains how real seed examples anchor a teacher model’s generation prompt so synthetic data matches the target data’s style.

Key terms: style anchoring, real seed examples, teacher model, distribution alignment

3.6.3 Step 3: Calling the teacher model to generate the candidates (one call per category)

This section explains why synthetic candidates are generated with one teacher-model call per category and how the returned responses are parsed into valid examples.

Key terms: API call per category, category balance, target count, JSON array, parse_candidates(), parse_candidate(), label, source="synthetic"

3.6.4 Step 4: Quality gate

This section explains how to filter synthetic financial-text candidates with an LLM judge before using them as training data.

Key terms: quality gate, Fidelity, Format, Domain safety, LLM-as-judge, PASS, candidate pool

3.6.5 Step 5: Distribution alignment check

This section explains why synthetic examples should be checked against real data before mixing and how to use a lightweight similarity score to decide what to do next.

Key terms: distribution alignment, synthetic distribution, real distribution, centroid cosine similarity, sentence transformer, mean embedding, synthetic proportion

3.6.6 Step 6: Mix with real data and save with a version manifest

This step caps the synthetic share when mixing training data, keeps evaluation data real, and saves a version manifest alongside the datasets.

Key terms: 30% threshold, n_synth, n_real, hard limit, evaluation split, real examples, teacher model's preferred style, version manifest, manifest.json, output_dir

3.7 Golden test sets and held-out evaluation

This section establishes the golden test set as the pre-shipping gate for showing that an adapted model improves on the base model.

Key terms: golden test set, base model, regression

3.7.1 What goes in a golden test set

The section defines the properties and practical size of a golden test set for evaluating an adapted model against its production workload.

Key terms: golden test set, representative, frozen, held out, reference answer, leakage, failure modes, 50 to 200 examples

3.7.2 Building your first golden test set

This section gives a four-step process for building a task-specific, expert-verified, versioned golden test set.

Key terms: user-facing tasks, slice, production logs, PII redaction, edge cases, domain expert, reference answer, rationale, borderline cases, JSONL

3.7.3 The red team companion set

This subsection explains why a separate red-team set is needed alongside the golden set and gives a starting size and maintenance rule for it.

Key terms: golden set, red-team set, prompt-injection attempts, jailbreaks, escalation to a human, training set, red-team exercises, attack categories

3.7.4 Common mistakes

This section identifies four mistakes that undermine a golden test set and explains how to preserve its value as an evaluation gate.

Key terms: golden set, train and eval, synthetic-data pipeline, edge cases, per-slice scores, critical edge-case slice, reference answer, reference rationale, auditable

3.7.5 Where the golden set fits in the adaptation lifecycle

This section places training data and the golden set on separate paths in the adaptation lifecycle so the model can be adapted and then evaluated independently.

Key terms: training set, human-labeled seed examples, synthetic generation, golden set, domain-expert reference answer, production logs, past tickets, adaptation lifecycle

3.8 Removing personal data before training

This section explains why personal data must be removed before fine-tuning and how to check for both obvious identifiers and identifying combinations of attributes.

Key terms: personal data, training distribution, right-to-erasure, PII, structured identifiers, indirect identifiers, human review, automated scrubbing, audit trail, annotator

3.8.1 Data retention and the right to be forgotten

This section explains how to honor personal-data deletion requests for fine-tuning data and set enforceable training-data retention deadlines.

Key terms: right to erasure, fine-tuning dataset, retraining, lineage tracking, training example, individual identifier, dataset version, retraining obligation, retention deadline, dataset manifest

3.8.2 Encryption in transit and at rest

This subsection explains how to protect training data with encryption and access controls throughout its lifecycle.

Key terms: encryption at rest, encryption in transit, AES-256, TLS 1.2+, least privilege, publicly accessible cloud storage bucket, non-public access

3.9 Summary

3.10 References

This section lists references for the chapter without explaining new concepts or procedures.

Where this is used later

Concepts explained in this chapter

← Chapter 2: How to do model adaptation Chapter 4: In-context learning, few-shot, and RAG →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.