liveBook · Manning

Synthetic data

As explained in LLM Customization and Fine-Tuning

Artificially generated examples used for training or evaluation.

Synthetic data consists of artificially generated examples used to supplement a dataset. For model adaptation, a teacher model can expand a small set of high-quality real examples into labeled variations, especially for rare scenarios that are difficult to collect. Generated examples need verification before they are mixed with real data. section 3.4.3 section 3.9

Start with clean human-labeled examples: they anchor the teacher’s output to the desired style, vocabulary, and domain. Without those references, generated examples can reflect the base model’s patterns rather than the target domain. Use synthetic generation when evaluation reveals a coverage gap, not simply to increase volume. section 3.1 section 3.4

A practical generation pipeline follows six steps:

  • Load high-quality seed examples and group them by category. section 3.6.1
  • Build style-anchored prompts using real examples from the target category, then call a teacher model for each category. section 3.4.3 section 3.9
  • Apply an LLM-as-judge quality gate for fidelity, format, and domain safety. section 3.9
  • Check distribution alignment using centroid cosine similarity, then mix accepted examples with real data. Use a maximum of 30% synthetic data as a first-run heuristic. section 3.9 section 3.4.3

Judge quality against the task. An invented hostname may be harmless filler in a ticket-classification example, but a summary that invents a host or root cause teaches the confident error the system should avoid. Synthetic data complements rather than replaces real examples; too much can narrow the model’s picture of the real distribution. section 3.4.3

Expanding sentiment examples

A demonstration loads 50–60 examples from the AllAgree split of Financial PhraseBank, where all 16 annotators agreed on each label. It groups seeds by sentiment so a prompt for negative examples can show three real negative examples rather than random samples. The public dataset demonstrates the pipeline without exposing a private support queue. section 3.6.1

Table 3.2 Different data acquisition strategies and what is best used for
Strategy
Relative cost
Label quality
Best used for
Human annotation
High
Highest
Seed data, evaluation sets, high-stakes edge cases
Weak supervision
Low
Moderate
High-volume categories with clear labeling rules
Synthetic generation
Medium
High (with gate)
Rare categories, gap-filling, controlled variation
Production log mining
Low
Variable
Distribution alignment, temporal freshness

Compare synthetic generation with human annotation, weak supervision, and production log mining by cost, label quality, and best use. section 3.4

Combine acquisition methods according to the budget and quality target: human labels provide anchors and evaluation examples, weak supervision scales clear rules, and production logs help maintain distribution alignment. Scrub personal data before it reaches annotators or training infrastructure; removing it from a dataset after training does not remove it from the model. section 3.4 section 3.9

How LLM Customization and Fine-Tuning covers it

How deep it goes
Details seed loading, category-anchored prompting, quality gating, distribution checks, and mixing limits in a six-step pipeline.
Examples
Uses Financial PhraseBank sentiment seeds and IT-support ticket categories to illustrate generation and task-specific verification.
What it assumes
Starts with clean human-labeled seed examples rather than requiring a large existing training set.
What you can do after
Apply it via a teacher LLM to expand category-matched seeds, verify generated examples, and mix them with real data.
How it connects
Connects synthetic generation to human annotation, weak supervision, production log mining, fine-tuning, and distribution alignment.

Also explained in

Compare how 24 books explain synthetic data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.