Chapter 3: What data do I need for model adaptation?
This chapter introduction explains why having enterprise data is not the same as having data suitable for model adaptation, using IT support records to show why quality and representativeness matter.
After this chapter you will be able to
- AI-ready data and model behavior. Explain why the quality and representativeness of training data matter more than simply having a large collection of records: a model learns from what the data teaches it, including its inconsistencies and gaps.
- Data readiness and clean examples. Describe the starting levels of the data readiness spectrum and what makes an input/output pair a clean example. Explain why a Level 2 dataset can be enough to begin adaptation.
- Human examples as seeds. Justify starting with human-written examples, even when only ten or twenty are available, before generating synthetic data.
- Data defects and model accuracy. Explain why a model can learn incorrect patterns from training data and describe how ordinary data defects reduced precision in the cited experiment.
- Quality effects beyond accuracy. Describe how low-quality or unrepresentative adaptation data affects user trust, fairness, and training effort, as well as accuracy.
- Training data provenance and poisoning. Distinguish the narrow poisoning result from the amount of good data needed for ticket triage, and explain why training-data provenance matters.
- Initial data needs versus later needs. Explain why Maya’s ticket-triage team can start with roughly 200 to 300 examples rather than waiting for 10,000, and why a few hundred examples may not be enough for a mature model.
- Starting Model and Data Needs. Distinguish base models from instruct models and explain how that difference affects the number of examples needed for adaptation.
- Training and Inference Prompt Match. Explain why examples for an instruct model still include the instruction the application will send at inference time.
- Diminishing accuracy gains. Explain how accuracy typically changes as training examples are added and why the curve levels off.
- Early example selection. Justify putting disproportionate effort into selecting and cleaning the first 200 to 300 training examples rather than simply assembling a much larger dataset.
- Category Distribution Audit. Describe a rough audit of whether training data reflects production: group tickets by scenario, compare each category’s share in the training set and a sample of real tickets, and flag underrepresented categories.
- High Risk Coverage Gaps. Explain why resolved-ticket exports can underrepresent important incidents and why a weakly covered, high-risk category should be fixed rather than simply increasing the overall example count.
- Disproportionate edge-case risk. Explain why a dataset dominated by common queries can produce a model that performs well overall but remains unsafe to deploy on rare, high-risk cases.
- Hard-variant dataset planning. Apply the section’s rule of adding at least three hard variants per ten common examples, and explain why it favors depth across a manageable number of scenarios over shallow coverage of many.
- User population representativeness. Explain why training on relevant query topics can still produce a mismatch when the examples come from the wrong users.
- Deployment dimension audit. Describe how to audit training data for deployment-relevant biases and what sufficient, rather than equal, representation means.
- Early Example Value. Explain why teams should prioritize a small set of high-quality, representative examples before collecting more fine-tuning data.
- Training Seed and Test Set. Distinguish the small labeled seed used to anchor training from the golden test set reserved for measuring progress.
- Plateau Driven Data Selection. Describe how to respond when adding examples stops improving accuracy and how to check for that plateau.
- Complementary acquisition strategies. Explain why an enterprise dataset may combine human annotation, weak supervision, synthetic generation, and production log mining, using their different roles in the IT support example.
- Starting strategy by data availability. Describe the recommended first move when a team has no labeled data, a large unlabeled corpus with clear rules, or production traffic from a deployed assistant.
- Expert Judgment in Ticket Labels. Explain why experienced support engineers provide reliable ground-truth labels for ticket categories and severity, and why fully human annotation is reserved for cases where label quality matters most.
- Clear Ticket Labeling Brief. Describe what a ticket-triage labeling brief must specify and explain how its clarity affects annotator agreement.
- Training and Evaluation Label Standards. Distinguish human-validated training labels from fully human-annotated labels, and justify the stricter standard for a held-out golden evaluation set.
- Labeling functions and voting. Trace how labeling functions assign candidate categories to tickets and how their votes determine which tickets receive labels or go to a human.
- Abstention rate calibration. Explain what the abstention rate reveals about labeling rules and how to use it with spot checks to revise them.
- Seed-Anchored Teacher Generation. Explain when synthetic examples help and how anchoring a teacher model to real seed examples keeps the generated data aligned with the target dataset.
- Task-Specific Synthetic Quality. Distinguish when an invented detail in a synthetic example is harmless from when it would teach an undesirable error.
- Real-Dominant Data Mix. Justify keeping synthetic examples a minority of the training mix and describe how the section applies its 30% cap.
- Production log representativeness. Explain why queries received by a deployed model or similar application are a particularly representative source of training examples.
- Production log mining pipeline. Describe the four steps for turning logged interactions into a sampled set of training examples, including why the final sample preserves categories.
- Reproducible Fine-Tuning Records. Explain why data versioning and experiment tracking matter for iterative fine-tuning: they let a team reproduce earlier results, diagnose regressions, and support an audit trail instead of relying on memory.
- Dataset and Run Linkage. Describe the minimum records that connect a training run to its data and result: a content hash stored with each dataset version, plus the dataset SHA-256, hyperparameters, and per-category evaluation scores logged for the run. Explain that tracking tools can also link broader training context to dataset and checkpoint identifiers.
- Reproducible experiment records. Explain why experiment logging must capture a training run’s inputs and information across data, configuration, infrastructure, and results.
- Tracking tool deployment choice. Distinguish W&B's hosted service from self-hosted MLflow and explain why organizational data policy, rather than tracking capability, drives the choice described here.
- Source and annotation provenance. Describe what source and annotation provenance record for a training example and how annotation records help investigate quality issues or deletion requests.
- Transformation records and PII audits. Explain what transformation provenance records and why PII scrubbing needs a specific audit trail.
- Manifest-linked lineage chain. Explain how storing lineage with a dataset manifest and linking versions preserves the path from a model checkpoint to original sources.
- Training-data regression diagnosis. Trace how hashes, manifests, and example-level provenance help distinguish changed data, annotation quality issues, and lost category coverage during a regression.
- Teacher-Generated Training Examples. Explain how a stronger LLM generates labeled examples in the style and domain of existing real data, and why the examples are verified before being mixed with seed data.
- High-agreement seed selection. Describe the first step as loading roughly 50 to 60 examples from Financial PhraseBank’s AllAgree split, where all 16 annotators agreed on each label, to use as style anchors.
- Category-matched style anchoring. Explain why the seeds are grouped by sentiment category before prompt building: the teacher can see real examples matching the category it must generate, which yields more consistent output than random anchors, especially for minority classes.
- Seed Example Style Anchoring. Explain how placing two or three real examples from the target category in a generation prompt steers the teacher model’s style, and why that matters for synthetic training data.
- Per-category generation calls. Explain why the pipeline makes a separate teacher-model API call for each category instead of one call for all categories.
- Teacher response parsing. Describe how teacher-model responses are turned into valid synthetic candidates despite common response-format and labeling problems.
- Financial Candidate Quality Rubric. Explain why plausible-sounding teacher-generated financial text needs checking and describe the three dimensions the second LLM scores: label fidelity, financial-news format, and plausibility of financial facts.
- Strict Pass And Pool Filtering. State the pass rule and explain why rejecting a portion of generated candidates is desirable rather than expecting the teacher to produce only perfect examples.
- Synthetic data distribution mismatch. Explain why checking synthetic examples against real data matters before mixing them for training.
- Centroid similarity mixing decisions. Describe how the centroid cosine similarity check compares real and synthetic pools, and apply the section’s score ranges to a mixing decision.
- Synthetic Data Share Limit. Explain how the pipeline limits synthetic examples to 30% of the combined dataset and calculate the maximum number allowed for a given number of real examples.
- Real-Only Evaluation Split. Justify why the evaluation split contains only real examples rather than synthetic ones.
- Golden test set gate. Explain why an adapted model should be compared with the base model on tasks the organization cares about before it ships, and what a golden test set makes possible.
- Golden test set properties. Explain why a golden test set must be representative, frozen, and held out: its inputs should reflect production traffic, its inputs and reference answers should stay stable across comparisons, and its examples must not enter training, prompt iteration, or synthetic-data generation.
- Task slices and representative cases. Describe how to choose the tasks and examples in a golden test set: give each major user-facing task its own slice, then include realistic common cases and consequential edge cases.
- Expert-verified reference answers. Explain why a domain expert should write or verify each reference answer and record its rationale, including when experts might disagree.
- Frozen and versioned test set. Describe how to save, deliberately update, and track the golden set so evaluations across model iterations remain comparable.
- Red-team set purpose and contents. Distinguish the red-team set from the golden set and describe the adversarial or safety-critical inputs it contains, including requests the assistant should refuse or escalate to a human.
- Train–evaluation separation. Explain why golden examples must not seed synthetic training data and why tooling should keep training and evaluation sets disjoint.
- Business-critical slice scores. Explain why an overall golden-set score is insufficient and how per-slice scores inform a shipping decision.
- Independent golden set evaluation. Explain where golden-set cases come from, how they are prepared, and why they remain separate from training data.
- Training Data Privacy Risk. Explain why personal data must be removed before training and why deleting it from a dataset after training is insufficient.
- Structured and Indirect Identifiers. Distinguish structured identifiers from combinations of attributes that can indirectly identify someone, and describe why both need checking.
- Pretraining Scrubbing Safeguards. Describe the section's practical scrubbing sequence and when it must occur relative to annotation, sharing, and training storage.
- Fine-Tuning Data Erasure. Explain why deleting a person's database record is not enough when their data was used for fine-tuning, and describe what must happen to the affected examples and model.
- Training Example Lineage. Justify recording the source, collection date, and individual identifier for each training example so examples affected by an erasure request can be found.
- Deletion and Retention Controls. Describe how to track a deletion request through dataset cleanup and model replacement, and how to set and document a training-data retention deadline.
- Encryption across training data locations. Explain that training data should be encrypted at rest and in transit wherever it lives, including source systems, staging, training infrastructure, and backups.
- Least privilege and storage access. Explain why access to training data should be restricted by least privilege and why cloud storage must be verified as non-public, even when encryption is enabled.
What it builds on
- How each technique changes the attack surface (chapter 2) · Fine-tuning changes the security attack surface.
Section by section
Introduction
This chapter introduction explains why having enterprise data is not the same as having data suitable for model adaptation, using IT support records to show why quality and representativeness matter.
- AI-ready data and model behavior: Explain why the quality and representativeness of training data matter more than simply having a large collection of records: a model learns from what the data teaches it, including its inconsistencies and gaps.
Why it matters: It helps you judge whether existing records are suitable before treating their volume as an advantage. - Limits of existing support records: Describe why Maya’s existing IT support records cannot simply be assumed ready for training, despite their quantity and variety.
Why it matters: Maya’s records show why a team must examine its existing data rather than assume that years of tickets are ready to use.
Key terms: AI-ready data, high-quality, representative data, data quality, ticketing system, resolved threads, knowledge base, support chat logs
3.1 Data readiness spectrum
This section explains how to judge whether an adaptation dataset is ready, why to start with human-written examples, and what to check before expanding the data.
- Data readiness and clean examples: Describe the starting levels of the data readiness spectrum and what makes an input/output pair a clean example. Explain why a Level 2 dataset can be enough to begin adaptation.
Why it matters: A modest dataset can be a useful starting point if its examples are suitable for the task. - Human examples as seeds: Justify starting with human-written examples, even when only ten or twenty are available, before generating synthetic data.
Why it matters: The examples used to start generation influence whether expanded data reflects your domain. - Data preparation decision order: Trace the checks in the section's data-preparation decision chart, from labeled examples through quality, sufficiency, and production representativeness.
Why it matters: These checks help identify whether the immediate data problem is availability, quality, quantity, or fit to production.
Key terms: data readiness spectrum, Level 1, Level 2, clean example, base, human seeds, synthetic generation, base model, labeled examples, data quality
3.2 Why data quality is important
This section explains how flawed or untrusted adaptation data can damage model performance, fairness, reliability, and security.
- Data defects and model accuracy: Explain why a model can learn incorrect patterns from training data and describe how ordinary data defects reduced precision in the cited experiment.
Why it matters: Checking the data can address an accuracy problem that more training would not fix. - Quality effects beyond accuracy: Describe how low-quality or unrepresentative adaptation data affects user trust, fairness, and training effort, as well as accuracy.
Why it matters: A data problem can affect adoption, equitable service, and project cost—not just an accuracy metric. - Training data provenance and poisoning: Distinguish the narrow poisoning result from the amount of good data needed for ticket triage, and explain why training-data provenance matters.
Why it matters: A large training set does not remove the need to verify where its documents came from.
Key terms: adaptation data, imbalanced labels, swapped fields, truncated values, precision, trust and reliability, bias and fairness, accuracy, training efficiency and cost, poisoning
3.3 How much data do you need?
The section explains why a few hundred high-quality, representative examples can support an initial fine-tuning test, while later improvement may require substantially more data.
- Initial data needs versus later needs: Explain why Maya’s ticket-triage team can start with roughly 200 to 300 examples rather than waiting for 10,000, and why a few hundred examples may not be enough for a mature model.
Why it matters: It helps the team avoid delaying its first experiment without mistaking that experiment’s data needs for its long-term needs. - Quality and evaluation first: Describe how example quality, representativeness, and early evaluation should guide decisions about collecting more training data.
Why it matters: It makes data collection responsive to observed model performance rather than an arbitrary ticket count.
Key terms: first useful run, quality ceiling, edge cases, 80/20 split, quality before quantity, representative data, evaluate early
3.3.1 Base models versus instruct models
This section explains how the starting model affects training-data needs and why training examples must match the prompts used at inference.
- Starting Model and Data Needs: Distinguish base models from instruct models and explain how that difference affects the number of examples needed for adaptation.
Why it matters: Your starting model determines whether scarce examples can focus on the domain or must also establish instruction-following. - Training and Inference Prompt Match: Explain why examples for an instruct model still include the instruction the application will send at inference time.
Why it matters: Matching the deployed prompt prevents an avoidable accuracy loss that may otherwise be difficult to diagnose.
Key terms: base model, instruct model, instruction-following, instruction, user turn, assistant turn, inference time, chat template
3.3.2 How accuracy responds to data volume
This section explains why fine-tuning accuracy gains diminish as training data grows and why the first examples deserve careful selection.
- Diminishing accuracy gains: Explain how accuracy typically changes as training examples are added and why the curve levels off.
Why it matters: The shape of the curve helps you judge whether collecting more examples is likely to improve accuracy enough to be worthwhile. - Early example selection: Justify putting disproportionate effort into selecting and cleaning the first 200 to 300 training examples rather than simply assembling a much larger dataset.
Why it matters: It directs limited data-collection effort toward examples most likely to improve the model.
Key terms: accuracy, training examples, plateau, marginal gain, clean, representative, diverse
3.3.3 Does your training data reflect your production distribution?
This section shows how to compare training tickets with the production queue and prioritize gaps in important scenarios.
- Category Distribution Audit: Describe a rough audit of whether training data reflects production: group tickets by scenario, compare each category’s share in the training set and a sample of real tickets, and flag underrepresented categories.
Why it matters: A rough comparison tells you which kinds of work your training set may be missing. - High Risk Coverage Gaps: Explain why resolved-ticket exports can underrepresent important incidents and why a weakly covered, high-risk category should be fixed rather than simply increasing the overall example count.
Why it matters: A dataset can contain plenty of examples yet offer little practice on the incidents where mistakes matter most.
Key terms: production sample, training set, scenario categories, distributions, resolved-ticket exports, P1 outages, high-risk, overall example counts
3.3.4 Edge cases matter disproportionately
This section explains why fine-tuning datasets need deliberately sourced hard cases, boundary conditions, and error-handling examples rather than simply mirroring common production queries.
- Disproportionate edge-case risk: Explain why a dataset dominated by common queries can produce a model that performs well overall but remains unsafe to deploy on rare, high-risk cases.
Why it matters: A strong overall accuracy figure can hide failures on the cases the business can least afford to get wrong. - Hard-variant dataset planning: Apply the section’s rule of adding at least three hard variants per ten common examples, and explain why it favors depth across a manageable number of scenarios over shallow coverage of many.
Why it matters: Planning for difficult variants early makes a limited data budget more useful than spreading it thinly across too many scenarios. - Realistic error-handling examples: Describe the error-handling situations that training examples should include and the kind of response those examples should demonstrate.
Why it matters: A model must handle incomplete or contradictory real-world requests, not just cases where everything goes right.
Key terms: example distribution, production queries, edge cases, business risk, hardest variant, boundary conditions, distinct scenarios, target dataset, error handling, incident thread
3.3.5 Diversity across user types, conditions, and variations
Training data must reflect the users and conditions a model will encounter, not just the topics of their queries.
- User population representativeness: Explain why training on relevant query topics can still produce a mismatch when the examples come from the wrong users.
Why it matters: Matching topics is not enough if the model will serve people who ask and interpret questions differently. - Deployment dimension audit: Describe how to audit training data for deployment-relevant biases and what sufficient, rather than equal, representation means.
Why it matters: A targeted audit helps reveal missing ways that intended users frame the same underlying question. - Market regime variation: Explain why training data should include variation in market conditions over time.
Why it matters: A model may respond poorly to changed conditions even when it has encountered the underlying concepts.
Key terms: representativeness, user population, response depth, systematic biases, seniority level, functional role, market context, sufficient representation, market conditions, temporal variation
3.3.6 Diminishing returns: Why the first 100 examples do most of the work
This section explains how diminishing returns should shape fine-tuning data collection and evaluation.
- Early Example Value: Explain why teams should prioritize a small set of high-quality, representative examples before collecting more fine-tuning data.
Why it matters: It helps teams spend their initial labeling effort where it is likely to have the greatest effect. - Training Seed and Test Set: Distinguish the small labeled seed used to anchor training from the golden test set reserved for measuring progress.
Why it matters: A separate test set makes changes in measured performance meaningful as training data grows. - Plateau Driven Data Selection: Describe how to respond when adding examples stops improving accuracy and how to check for that plateau.
Why it matters: Checking failures and progress prevents wasted collection of examples that repeat what the model already handles.
Key terms: high-quality, representative examples, accuracy gain, golden seed, golden test set, hold out, failure cases, missing coverage, held-out golden set, saturated the current distribution
3.4 Data acquisition strategies
This section explains how to combine data acquisition strategies and choose a starting approach based on available data, effort, and quality needs.
- Complementary acquisition strategies: Explain why an enterprise dataset may combine human annotation, weak supervision, synthetic generation, and production log mining, using their different roles in the IT support example.
Why it matters: Combining methods lets a team cover different data needs without relying on one method for everything. - Starting strategy by data availability: Describe the recommended first move when a team has no labeled data, a large unlabeled corpus with clear rules, or production traffic from a deployed assistant.
Why it matters: Starting with the resources already available avoids unnecessary up-front labeling. - Quality anchored data mix: Describe the suggested starting composition for an enterprise dataset and when to add synthetic examples.
Why it matters: This gives the team a bounded starting mix rather than an invitation to generate examples indiscriminately.
Key terms: human annotation, weak supervision, synthetic generation, production log mining, cost-quality-speed space, seed dataset, quality gate, well-labeled examples, hard variants, synthetic proportion
3.4.1 Human annotation
This section explains when to use human annotation, how a clear labeling brief improves agreement, and why the golden evaluation set needs independent human labels.
- Expert Judgment in Ticket Labels: Explain why experienced support engineers provide reliable ground-truth labels for ticket categories and severity, and why fully human annotation is reserved for cases where label quality matters most.
Why it matters: Expert labels are especially valuable when a ticket’s apparent symptoms or tone could lead to the wrong classification. - Clear Ticket Labeling Brief: Describe what a ticket-triage labeling brief must specify and explain how its clarity affects annotator agreement.
Why it matters: A clear brief makes the labels consistent enough to be useful, rather than merely numerous. - Training and Evaluation Label Standards: Distinguish human-validated training labels from fully human-annotated labels, and justify the stricter standard for a held-out golden evaluation set.
Why it matters: An independently labeled test set can reveal errors that a test set built like the training data might conceal.
Key terms: human annotation, ground truth, severity definitions, labeling brief, label set, disambiguation rules, worked examples, Cohen’s kappa, human-validated, fully human-annotated
3.4.2 Weak supervision
This section explains how weak supervision uses noisy labeling rules and majority voting to label large ticket corpora, then uses abstentions and human review to improve the results.
- Weak supervision tradeoff: Explain when weak supervision is useful and why its greater label volume and coverage can justify a higher error rate than human labeling.
Why it matters: It helps you decide whether fast, inexpensive labeling is worth the loss in label accuracy. - Labeling functions and voting: Trace how labeling functions assign candidate categories to tickets and how their votes determine which tickets receive labels or go to a human.
Why it matters: Voting lets the pipeline use imperfect rules without trusting any one rule to label every ticket correctly. - Abstention rate calibration: Explain what the abstention rate reveals about labeling rules and how to use it with spot checks to revise them.
Why it matters: The abstention rate helps you catch both rules that miss too many tickets and rules that label too confidently. - LLM judge as voter: Describe how an LLM-as-judge can contribute votes and why it should receive the human annotators’ brief.
Why it matters: A shared brief makes the model’s votes more consistent with the team’s labeling criteria.
Key terms: weak supervision, unlabeled corpus, labeling rules, labeling functions, majority vote, non-abstaining functions, abstention rate, spot check, weak coverage, LLM-as-judge
3.4.3 Synthetic data from teacher models
This section explains how to expand a small real dataset with teacher-generated examples while preserving its style, checking examples against the task, and keeping real data dominant.
- Seed-Anchored Teacher Generation: Explain when synthetic examples help and how anchoring a teacher model to real seed examples keeps the generated data aligned with the target dataset.
Why it matters: Topically relevant examples can still teach the student the wrong patterns if they look unlike the real data. - Task-Specific Synthetic Quality: Distinguish when an invented detail in a synthetic example is harmless from when it would teach an undesirable error.
Why it matters: A useful quality check catches errors that matter to the trained behavior without rejecting harmless invented context. - Real-Dominant Data Mix: Justify keeping synthetic examples a minority of the training mix and describe how the section applies its 30% cap.
Why it matters: Keeping real examples dominant helps retain the breadth needed for rare but important inputs.
Key terms: teacher model, seed dataset, anchoring, distributional shift, quality gate, hallucination, classification, summarization, synthetic share, 30% cap
3.4.4 Production log mining
This section explains why production logs are a representative source of training examples and outlines a four-step process for mining them safely and usefully.
- Production log representativeness: Explain why queries received by a deployed model or similar application are a particularly representative source of training examples.
Why it matters: Real user queries help training data reflect the requests the application actually receives. - Production log mining pipeline: Describe the four steps for turning logged interactions into a sampled set of training examples, including why the final sample preserves categories.
Why it matters: These steps make logged examples safer to review and help the selected data retain important query types.
Key terms: production log mining, production environment, query and response pairs, PII scrubbing, deduplication, normalized query text, stratified sampling, category distribution
3.5 Data versioning and experiment tracking
The section explains how versioned data, training-run records, and example-level provenance make fine-tuning reproducible and easier to diagnose.
- Reproducible Fine-Tuning Records: Explain why data versioning and experiment tracking matter for iterative fine-tuning: they let a team reproduce earlier results, diagnose regressions, and support an audit trail instead of relying on memory.
Why it matters: Good records let someone investigate a model change without reconstructing an old training run from memory. - Dataset and Run Linkage: Describe the minimum records that connect a training run to its data and result: a content hash stored with each dataset version, plus the dataset SHA-256, hyperparameters, and per-category evaluation scores logged for the run. Explain that tracking tools can also link broader training context to dataset and checkpoint identifiers.
Why it matters: Linking these identifiers and results answers which data and settings produced a particular checkpoint. - Example-Level Data Provenance: Describe the provenance to record when each example enters the pipeline, including its source, annotation, and transformations, so its lineage can be checked later.
Why it matters: Per-example lineage helps a team establish where training data came from and how it was prepared.
Key terms: data versioning, experiment tracking, regressions, regulatory audit trail, content-hash, manifest, dataset SHA-256, hyperparameter configuration, per-category evaluation scores, model checkpoint
3.5.1 Experiment logging
This subsection defines what an LLM fine-tuning experiment record must capture so runs can be reproduced and compared.
- Reproducible experiment records: Explain why experiment logging must capture a training run’s inputs and information across data, configuration, infrastructure, and results.
Why it matters: A complete record lets you tell what changed between runs and repeat a result.
Key terms: experiment logging, experiment record, data, configuration, infrastructure, results
3.5.2 Choosing a tracking tool
This section explains what W&B and MLflow track, how to enable their logging in SFTTrainer, and how data policy affects the choice between them.
- Experiment tracking capabilities: Explain that W&B and MLflow record training settings and metrics to help compare runs, spot regressions, and reproduce results, but do not automatically version the input dataset.
Why it matters: Logged runs are useful for comparison, but they are not a complete record of the data used. - Trainer logging configuration: Describe how to enable W&B or MLflow logging through SFTTrainer's SFTConfig.
Why it matters: A configuration choice connects the training run to either tracking tool. - Tracking tool deployment choice: Distinguish W&B's hosted service from self-hosted MLflow and explain why organizational data policy, rather than tracking capability, drives the choice described here.
Why it matters: The tracking tool must fit where your organization permits training metadata to go.
Key terms: hyperparameters, training metrics, dataset versioning, SFTTrainer, SFTConfig, report_to, W&B, MLflow, training metadata, data governance
3.5.3 Data lineage
This section explains how per-example data lineage and dataset versioning make training data auditable and help diagnose model regressions.
- Source and annotation provenance: Describe what source and annotation provenance record for a training example and how annotation records help investigate quality issues or deletion requests.
Why it matters: These records let you trace a suspect example or annotation batch instead of guessing where a problem began. - Transformation records and PII audits: Explain what transformation provenance records and why PII scrubbing needs a specific audit trail.
Why it matters: A recorded transformation history makes it possible to audit changes and demonstrate that personal data was removed. - Manifest-linked lineage chain: Explain how storing lineage with a dataset manifest and linking versions preserves the path from a model checkpoint to original sources.
Why it matters: Linked manifests preserve a traceable history even as training datasets gain new versions. - Training-data regression diagnosis: Trace how hashes, manifests, and example-level provenance help distinguish changed data, annotation quality issues, and lost category coverage during a regression.
Why it matters: This narrows a regression to a plausible cause before you change the model or dataset. - Production-only distribution drift: Distinguish a regression seen only in production from one reflected in held-out testing, and describe the response given here.
Why it matters: The held-out-versus-production comparison helps avoid misdiagnosing changing user queries as a defect in the old training set.
Key terms: lineage, source provenance, annotation provenance, task brief version, agreement scores, transformation provenance, PII scrubbing, audit trail, dataset manifest, lineage chain
3.6 Generating synthetic training data
Introduces synthetic training data generation as using a stronger teacher LLM to create examples that are verified before joining real seed data.
- Teacher-Generated Training Examples: Explain how a stronger LLM generates labeled examples in the style and domain of existing real data, and why the examples are verified before being mixed with seed data.
Why it matters: This distinguishes generated candidates from examples ready to use for training.
Key terms: synthetic generation, teacher, labeled examples, seed data
3.6.1 Step 1: Load seed examples from Hugging Face
This subsection chooses public, high-agreement Financial PhraseBank examples and groups them by sentiment to serve as style anchors for synthetic-data generation.
- Cross-domain example choice: Justify using Financial PhraseBank instead of Maya’s private tickets or the book’s familiar IT-support corpus: a public, licensed dataset protects private data, and success in a nonoverlapping domain provides stronger evidence that the pipeline generalizes.
Why it matters: The choice of dataset makes the example usable without private tickets and tests whether the approach carries over to another domain. - High-agreement seed selection: Describe the first step as loading roughly 50 to 60 examples from Financial PhraseBank’s AllAgree split, where all 16 annotators agreed on each label, to use as style anchors.
Why it matters: The selected seeds provide trusted examples for the prompts built in the next step. - Category-matched style anchoring: Explain why the seeds are grouped by sentiment category before prompt building: the teacher can see real examples matching the category it must generate, which yields more consistent output than random anchors, especially for minority classes.
Why it matters: Grouping seeds by category lets each generation prompt show the teacher examples relevant to the output it is being asked to produce.
Key terms: Financial PhraseBank, public, licensed dataset, domain-agnostic, AllAgree split, style anchors, sentiment category, category-matched seeds, style anchoring, minority classes
3.6.2 Step 2: Build the prompt with style anchoring
This section explains how real seed examples anchor a teacher model’s generation prompt so synthetic data matches the target data’s style.
- Seed Example Style Anchoring: Explain how placing two or three real examples from the target category in a generation prompt steers the teacher model’s style, and why that matters for synthetic training data.
Why it matters: Style anchoring helps synthetic examples resemble the data your model will encounter in production.
Key terms: style anchoring, real seed examples, teacher model, distribution alignment
3.6.3 Step 3: Calling the teacher model to generate the candidates (one call per category)
This section explains why synthetic candidates are generated with one teacher-model call per category and how the returned responses are parsed into valid examples.
- Per-category generation calls: Explain why the pipeline makes a separate teacher-model API call for each category instead of one call for all categories.
Why it matters: Separate calls make generation more resilient and the intended category counts easier to control. - Teacher response parsing: Describe how teacher-model responses are turned into valid synthetic candidates despite common response-format and labeling problems.
Why it matters: Parsing keeps irregular teacher responses from being treated as ready-to-use training examples.
Key terms: API call per category, category balance, target count, JSON array, parse_candidates(), parse_candidate(), label, source="synthetic"
3.6.4 Step 4: Quality gate
This section explains how to filter synthetic financial-text candidates with an LLM judge before using them as training data.
- Financial Candidate Quality Rubric: Explain why plausible-sounding teacher-generated financial text needs checking and describe the three dimensions the second LLM scores: label fidelity, financial-news format, and plausibility of financial facts.
Why it matters: A fluent example can still teach the model a wrong label or an unrealistic financial claim. - Strict Pass And Pool Filtering: State the pass rule and explain why rejecting a portion of generated candidates is desirable rather than expecting the teacher to produce only perfect examples.
Why it matters: The all-dimensions rule keeps a strong-looking example with one serious flaw out of the training pool.
Key terms: quality gate, Fidelity, Format, Domain safety, LLM-as-judge, PASS, candidate pool
3.6.5 Step 5: Distribution alignment check
This section explains why synthetic examples should be checked against real data before mixing and how to use a lightweight similarity score to decide what to do next.
- Synthetic data distribution mismatch: Explain why checking synthetic examples against real data matters before mixing them for training.
Why it matters: A synthetic dataset can look useful yet make the training mix less representative of important real inputs. - Centroid similarity mixing decisions: Describe how the centroid cosine similarity check compares real and synthetic pools, and apply the section’s score ranges to a mixing decision.
Why it matters: The score turns a compatibility check into a practical decision about mixing or revising synthetic data.
Key terms: distribution alignment, synthetic distribution, real distribution, centroid cosine similarity, sentence transformer, mean embedding, synthetic proportion
3.6.6 Step 6: Mix with real data and save with a version manifest
This step caps the synthetic share when mixing training data, keeps evaluation data real, and saves a version manifest alongside the datasets.
- Synthetic Data Share Limit: Explain how the pipeline limits synthetic examples to 30% of the combined dataset and calculate the maximum number allowed for a given number of real examples.
Why it matters: The cap prevents synthetic examples from making up more of the combined dataset than this pipeline allows. - Real-Only Evaluation Split: Justify why the evaluation split contains only real examples rather than synthetic ones.
Why it matters: A real-only evaluation split keeps the test focused on the inputs the model is meant to handle in production. - Version Manifest Placement: Describe when and where the version manifest is saved.
Why it matters: The saved datasets and their version manifest remain together.
Key terms: 30% threshold, n_synth, n_real, hard limit, evaluation split, real examples, teacher model's preferred style, version manifest, manifest.json, output_dir
3.7 Golden test sets and held-out evaluation
This section establishes the golden test set as the pre-shipping gate for showing that an adapted model improves on the base model.
- Golden test set gate: Explain why an adapted model should be compared with the base model on tasks the organization cares about before it ships, and what a golden test set makes possible.
Why it matters: A shared evaluation gate makes a shipping decision depend on demonstrated improvement rather than impressions.
Key terms: golden test set, base model, regression
3.7.1 What goes in a golden test set
The section defines the properties and practical size of a golden test set for evaluating an adapted model against its production workload.
- Golden test set properties: Explain why a golden test set must be representative, frozen, and held out: its inputs should reflect production traffic, its inputs and reference answers should stay stable across comparisons, and its examples must not enter training, prompt iteration, or synthetic-data generation.
Why it matters: These properties keep evaluation results relevant to production and comparable across runs. - Golden set size and growth: Describe why careful curation matters more than size, give the suggested 50-to-200-example starting target for most enterprise uses, and explain how the set grows when new production failures appear.
Why it matters: The team can begin evaluating with a manageable set while preserving coverage of failures discovered later.
Key terms: golden test set, representative, frozen, held out, reference answer, leakage, failure modes, 50 to 200 examples
3.7.2 Building your first golden test set
This section gives a four-step process for building a task-specific, expert-verified, versioned golden test set.
- Task slices and representative cases: Describe how to choose the tasks and examples in a golden test set: give each major user-facing task its own slice, then include realistic common cases and consequential edge cases.
Why it matters: A single average score can hide poor performance on an important task or failure case. - Expert-verified reference answers: Explain why a domain expert should write or verify each reference answer and record its rationale, including when experts might disagree.
Why it matters: The test set cannot give trustworthy scores if its supposed correct answers are unreliable. - Frozen and versioned test set: Describe how to save, deliberately update, and track the golden set so evaluations across model iterations remain comparable.
Why it matters: Recorded test-set versions make it possible to interpret changes in model quality across runs.
Key terms: user-facing tasks, slice, production logs, PII redaction, edge cases, domain expert, reference answer, rationale, borderline cases, JSONL
3.7.3 The red team companion set
This subsection explains why a separate red-team set is needed alongside the golden set and gives a starting size and maintenance rule for it.
- Red-team set purpose and contents: Distinguish the red-team set from the golden set and describe the adversarial or safety-critical inputs it contains, including requests the assistant should refuse or escalate to a human.
Why it matters: A model can perform well on ordinary requests yet still fail on attacks or requests it must not fulfill. - Red-team set sizing and growth: State the suggested starting size for a production red-team set and describe how it should grow as new attack categories emerge.
Why it matters: A production safety check needs enough prompts to start with and must expand when new attacks appear.
Key terms: golden set, red-team set, prompt-injection attempts, jailbreaks, escalation to a human, training set, red-team exercises, attack categories
3.7.4 Common mistakes
This section identifies four mistakes that undermine a golden test set and explains how to preserve its value as an evaluation gate.
- Train–evaluation separation: Explain why golden examples must not seed synthetic training data and why tooling should keep training and evaluation sets disjoint.
Why it matters: An evaluation set cannot reliably test a model on examples it may have learned during training. - Preservation of failed examples: Explain why teams should not remove golden-set examples simply because the model fails them, even when they appear to be edge cases.
Why it matters: Keeping difficult examples visible prevents apparent improvement from being manufactured by changing the test. - Business-critical slice scores: Explain why an overall golden-set score is insufficient and how per-slice scores inform a shipping decision.
Why it matters: A reassuring average can mask failure on the cases the business can least afford to get wrong. - Reference answer rationales: Explain why each golden-set reference answer should have a brief written rationale.
Why it matters: A short explanation makes disputed answers easier to judge consistently later.
Key terms: golden set, train and eval, synthetic-data pipeline, edge cases, per-slice scores, critical edge-case slice, reference answer, reference rationale, auditable
3.7.5 Where the golden set fits in the adaptation lifecycle
This section places training data and the golden set on separate paths in the adaptation lifecycle so the model can be adapted and then evaluated independently.
- Training data preparation path: Describe how the training set starts with about 50 human-labeled seed examples, expands through synthetic generation, and is capped at 30% synthetic data.
Why it matters: It clarifies which examples are intended to change the model rather than measure its quality. - Independent golden set evaluation: Explain where golden-set cases come from, how they are prepared, and why they remain separate from training data.
Why it matters: Keeping evaluation cases out of training makes the golden set useful for judging model quality.
Key terms: training set, human-labeled seed examples, synthetic generation, golden set, domain-expert reference answer, production logs, past tickets, adaptation lifecycle
3.8 Removing personal data before training
This section explains why personal data must be removed before fine-tuning and how to check for both obvious identifiers and identifying combinations of attributes.
- Training Data Privacy Risk: Explain why personal data must be removed before training and why deleting it from a dataset after training is insufficient.
Why it matters: Prevention matters because cleaning a dataset later does not undo what the model learned. - Structured and Indirect Identifiers: Distinguish structured identifiers from combinations of attributes that can indirectly identify someone, and describe why both need checking.
Why it matters: Removing obvious fields alone can leave enough information to identify a person. - Pretraining Scrubbing Safeguards: Describe the section's practical scrubbing sequence and when it must occur relative to annotation, sharing, and training storage.
Why it matters: Checking and removing personal data before it spreads through the workflow reduces exposure and leaves a documented review trail.
Key terms: personal data, training distribution, right-to-erasure, PII, structured identifiers, indirect identifiers, human review, automated scrubbing, audit trail, annotator
3.8.1 Data retention and the right to be forgotten
This section explains how to honor personal-data deletion requests for fine-tuning data and set enforceable training-data retention deadlines.
- Fine-Tuning Data Erasure: Explain why deleting a person's database record is not enough when their data was used for fine-tuning, and describe what must happen to the affected examples and model.
Why it matters: A model may have been trained on data after the original record has been deleted. - Training Example Lineage: Justify recording the source, collection date, and individual identifier for each training example so examples affected by an erasure request can be found.
Why it matters: Lineage makes a deletion request actionable rather than an untraceable obligation. - Deletion and Retention Controls: Describe how to track a deletion request through dataset cleanup and model replacement, and how to set and document a training-data retention deadline.
Why it matters: Explicit records and deadlines keep deletion and retention obligations from being lost during model operations.
Key terms: right to erasure, fine-tuning dataset, retraining, lineage tracking, training example, individual identifier, dataset version, retraining obligation, retention deadline, dataset manifest
3.8.2 Encryption in transit and at rest
This subsection explains how to protect training data with encryption and access controls throughout its lifecycle.
- Encryption across training data locations: Explain that training data should be encrypted at rest and in transit wherever it lives, including source systems, staging, training infrastructure, and backups.
Why it matters: Protecting only the training environment leaves copies or transfers of sensitive data exposed. - Least privilege and storage access: Explain why access to training data should be restricted by least privilege and why cloud storage must be verified as non-public, even when encryption is enabled.
Why it matters: Strong encryption does not replace checking who can access the storage that holds training data.
Key terms: encryption at rest, encryption in transit, AES-256, TLS 1.2+, least privilege, publicly accessible cloud storage bucket, non-public access
3.9 Summary
3.10 References
This section lists references for the chapter without explaining new concepts or procedures.
Where this is used later
- Quality over quantity (chapter 6) · Returns to the importance of data quality relative to quantity.
- Data quality iterations (chapter 5) · Interpreting the need for data quality iterations during fine-tuning.
- Step 1: Prepare data (chapter 5) · Assumes the importance of data quality for model adaptation.
- Realistic expectations (chapter 6) · The role of training data volume in adaptation.
- Data requirements (chapter 6) · Earlier discussion of how much adaptation data is needed.
- Why low rank? (chapter 5) · Introduces the distinction between base and instruct models.
- Where an adapter beats prompting (chapter 5) · Assumes that training examples need diversity across relevant cases.
- Responsible AI and evaluation (chapter 5) · Diversity across user types and conditions motivates checking whether training examples represent relevant groups.
Concepts explained in this chapter
- synthetic dataArtificially generated examples used for training or evaluation.
- evaluation datasetA subset of examples reserved for evaluating a model rather than updating its parameters.
- quality gateA check that determines whether generated or collected examples meet quality requirements before they are accepted into a dataset.
- data acquisition strategyThe plan for collecting, selecting, and preparing data to support model adaptation.
- weak supervisionA data-labeling approach that produces training labels from noisy, indirect, or programmatic sources rather than manual annotation.
- PII scrubbingThe removal or masking of personally identifiable information from data.
- data annotationThe process of adding labels or other explanatory information to data so it can support model development or evaluation.
- dataset versioningThe practice of tracking and preserving distinct dataset versions for traceability and reproducibility.
- AI-ready dataData prepared and curated so it is suitable for training or adapting a model, rather than merely existing in an operational system.
- data qualityThe degree to which adaptation data is correct, relevant, and representative enough to support reliable model behavior.
- provenanceThe record of data’s origins, transformations, and movement through processing systems.
- personal dataPersonal data is information relating to an identified or identifiable individual.
- training datasetThe collection of examples used to fit or adapt a model during training.
- model adaptationModifying a pretrained model for a particular task, domain, or use case without training it from scratch.
- teacher modelA model whose outputs or behavior provide targets for training a student model during distillation.
- edge caseAn uncommon or difficult input situation that tests whether a model behaves correctly beyond routine scenarios.
- foundation modelThe original pretrained model whose weights are kept fixed while LoRA learns an additive adaptation.
- instruct modelA language model fine-tuned to reliably follow instructions or prompts.
- accuracyThe proportion of evaluated predictions or answers that are correct.
- data driftA change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.
- LLM-as-a-judgeAn evaluation setup in which a language model judges other model outputs against specified criteria.
- annotation provenanceInformation recording the human labeling history and review details associated with a data example.
- source provenanceInformation recording which source a data example came from and when it was collected or generated.
- transformation provenanceInformation recording the automated transformations applied to a data example.
- seed datasetA small set of initial examples used to condition or guide later synthetic generation.
- inter-rater reliabilityThe degree to which different annotators assign the same labels or answers to the same examples.
- fidelityA measure of whether a sentiment label matches the content of its sentence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
