Chapter 12: Synthetic data
The introduction explains why early RLHF depended on human data and why improving AI models made synthetic data an attractive way to expand post-training research.
After this chapter you will be able to
- Synthetic data adoption rationale. Explain why synthetic data became attractive as AI models improved and how its lower cost and easier iteration helped RLHF experiments proliferate.
- Synthetic data in post-training. Describe the roles synthetic data plays in modern post-training, including generating or modifying prompts, producing completions, creating preference feedback, and filtering outputs.
- Synthetic and human data roles. Distinguish where synthetic data has largely displaced human-written data from where human input remains important in instruction tuning, preferences, and evaluation.
- Model collapse and safeguards. Explain the model-collapse concern and why the section does not treat it as an inevitable result of using synthetic data at scale.
- Meaning of model distillation. Distinguish traditional teacher–student knowledge distillation from the broader use of the term in language-model post-training, where outputs from a stronger model train a smaller model.
- Two post-training uses. Describe the two common uses of distillation in post-training: supplying data across the process and transferring particular skills from a stronger model to a weaker one.
- Distillation as synthetic-data training. Explain why teacher–student distillation belongs in a chapter on synthetic data: a stronger model generates supervision, and a student model is trained on it. Describe how interest in this approach has grown as post-training shifts toward reasoning and agentic models.
- Student-sampled distillation objective. Describe how on-policy distillation samples completions from the current student and compares its next-token distribution with a fixed teacher on the resulting states. Explain why sampling from the student and placing its distribution on the left of the KL makes this a reverse-KL objective.
- Token-level teacher feedback. Explain how an RL implementation can use the negative per-token reverse-KL contribution as an advantage-like signal: a sampled token receives positive advantage when the teacher rates it above the student and negative advantage when the teacher rates it below the student.
- Prompt-weighted multiple teachers. Explain how multi-teacher on-policy distillation uses specialist models or earlier checkpoints as teachers and assigns each a contribution weight by prompt or task type.
- AI feedback and RLAIF scope. Distinguish AI feedback from RLAIF and describe where AI feedback fits in the RLHF pipeline. Explain that AI feedback can augment or replace human feedback, particularly for preference data collection and reward model training.
- Task-dependent feedback balance. Explain why the section does not recommend replacing all human feedback with AI feedback, and describe its practical advice for starting and scaling a pipeline.
- Noise and bias tradeoff. Distinguish the section’s rules of thumb for human versus synthetic preference data and explain the practical consequence of each.
- LLM judge reliability limits. Explain why using an LLM to judge responses raises calibration concerns, including inconsistent evaluations and self-preference bias.
- Choosing a judgment approach. Describe the alternatives mentioned for improving or specializing model judgments, and explain when the section says training your own judge may be warranted.
- Principle-guided instruction data. Trace how Constitutional AI creates instruction data by repeatedly critiquing and revising a model response against sampled principles, then fine-tuning on the final revision.
- Principle-guided preference data. Describe how a feedback model uses constitutional principles to choose between two completions, producing synthetic preference data that can be used in RLHF.
- Principle Based Alignment Extensions. Describe how related work uses written policies or principles to guide models, reward models, or human judgments, and explain the section’s caution about their status in RLHF recipes.
- Rubrics as training rewards. Explain why rubrics are useful for prompts without clearly verifiable answers and how they support reinforcement learning.
- Scalable rubric generation. Explain the cost of generating a rubric for every training prompt and the section's approach to reducing that burden.
What it builds on
- Why we need preference data (chapter 11) · Why RLHF needs preference data.
- Policy gradient algorithms (chapter 6) · The role of advantage signals in policy-gradient algorithms.
- Group relative policy optimization (chapter 6) · The earlier treatment of group relative policy optimization.
- Collecting preference data (chapter 11) · Introduces collecting preference data.
- (chapter intro) (chapter 5) · Introduces reward modeling as a stage of RLHF.
- Bias: Things to watch out for in data collection (chapter 11) · considering bias in preference data collection
- Chat templates and the structure of instructions (chapter 4) · The idea of a prompt and completion as an instruction example
Section by section
Introduction
The introduction explains why early RLHF depended on human data and why improving AI models made synthetic data an attractive way to expand post-training research.
- Early reliance on human data: Explain why human data was initially the only viable way to improve models with RLHF: it supplied high-quality responses and reliable, specific feedback.
Why it matters: This explains the starting point from which synthetic data changed RLHF. - Synthetic data adoption rationale: Explain why synthetic data became attractive as AI models improved and how its lower cost and easier iteration helped RLHF experiments proliferate.
Why it matters: Lower-cost, faster iteration helps explain why synthetic data became important to RLHF research.
Key terms: human data, RLHF, feedback data, synthetic data, post-training
12.1 The roles of synthetic data
The section explains how synthetic data is used in post-training, why its use has grown, and where human data remains essential.
- Synthetic data in post-training: Describe the roles synthetic data plays in modern post-training, including generating or modifying prompts, producing completions, creating preference feedback, and filtering outputs.
Why it matters: These varied roles explain why synthetic data has become central to post-training. - Synthetic and human data roles: Distinguish where synthetic data has largely displaced human-written data from where human input remains important in instruction tuning, preferences, and evaluation.
Why it matters: The value of synthetic data depends on the task; it does not remove the need for human ground truth. - Model collapse and safeguards: Explain the model-collapse concern and why the section does not treat it as an inevitable result of using synthetic data at scale.
Why it matters: It explains both a real risk of synthetic data and why that risk need not rule out its use. - Growth of synthetic datasets: Describe how more capable models enabled reliable synthetic-data generation and judging, alongside growth in the size and scope of post-training datasets.
Why it matters: The model-capability shift helps explain why synthetic data became practical at a much larger scale.
Key terms: synthetic data, training prompts, completions, AI feedback, preference data, instruction data, supervised fine-tuning, LLM-as-a-judge, ground-truth labels, model collapse
12.2 Distillation with synthetic data
This section explains how stronger models supply synthetic training data for broad post-training needs and for transferring specific skills to smaller models.
- Meaning of model distillation: Distinguish traditional teacher–student knowledge distillation from the broader use of the term in language-model post-training, where outputs from a stronger model train a smaller model.
Why it matters: This distinction makes clear which sense of distillation the section uses when discussing synthetic data. - Two post-training uses: Describe the two common uses of distillation in post-training: supplying data across the process and transferring particular skills from a stronger model to a weaker one.
Why it matters: The distinction shows that synthetic data can support both general post-training work and targeted improvements. - Teacher-data quality: Explain why selecting high-quality prompts and filtering teacher responses matter when generating synthetic training data.
Why it matters: The strength of the teacher alone does not remove the need to select useful training data.
Key terms: distillation, teacher–student knowledge distillation, stronger model, smaller model, data engine, post-training, preference data, specific skills, high-quality prompts, filtering responses
12.3 The path to on-policy, teacher–student distillation
This section explains why teacher–student distillation is part of the synthetic-data toolkit for language-model post-training and notes its renewed prominence.
- Distillation as synthetic-data training: Explain why teacher–student distillation belongs in a chapter on synthetic data: a stronger model generates supervision, and a student model is trained on it. Describe how interest in this approach has grown as post-training shifts toward reasoning and agentic models.
Why it matters: This connection explains why distillation appears here as a use of synthetic data rather than as an unrelated training topic.
Key terms: teacher–student knowledge distillation, synthetic data, supervision, student model
12.3.1 Adapting knowledge-distillation for LMs
12.3.2 From offline to on-policy distillation
This section explains why distillation moves from offline teacher-generated data to student-generated completions, and how teacher feedback on those completions can drive RL-style updates.
- Offline distillation limitations: Explain why offline knowledge distillation motivated on-policy variants: the teacher’s training sequences may differ from what the student generates at inference time, and a forward-KL objective can overestimate low-probability regions of the teacher distribution.
Why it matters: This explains the problem that on-policy distillation is meant to address. - Student-sampled distillation objective: Describe how on-policy distillation samples completions from the current student and compares its next-token distribution with a fixed teacher on the resulting states. Explain why sampling from the student and placing its distribution on the left of the KL makes this a reverse-KL objective.
Why it matters: Sampling from the student changes which states receive teacher feedback. - Token-level teacher feedback: Explain how an RL implementation can use the negative per-token reverse-KL contribution as an advantage-like signal: a sampled token receives positive advantage when the teacher rates it above the student and negative advantage when the teacher rates it below the student.
Why it matters: This connects distillation to RL updates while giving feedback on individual generated tokens.
Key terms: offline KD, distribution mismatch, forward KL, on-policy distillation, student policy, teacher policy, reverse-KL objective, token-level state, KD distance, reverse-KL distance
12.3.3 Modern OPD variants
This section describes modern on-policy distillation variants that use multiple teachers, combine distillation with other training signals, or let a model serve as its own teacher.
- Prompt-weighted multiple teachers: Explain how multi-teacher on-policy distillation uses specialist models or earlier checkpoints as teachers and assigns each a contribution weight by prompt or task type.
Why it matters: Prompt-dependent weights let a final model draw on different teachers for different tasks. - Combined distillation advantages: Describe the example of combining reverse KL with another form of advantage computation during OPD.
Why it matters: This shows that OPD's training signal need not be used alone. - Per-token tokenizer constraint: Explain why knowledge-distillation methods often require the student and teacher to share a tokenizer.
Why it matters: The form of teacher feedback can constrain which student and teacher models are paired. - Privileged self-distillation feedback: Describe the teacher's role in on-policy self-distillation as presented here.
Why it matters: This variant shows how verification can provide a teacher signal without a separate specialist teacher model.
Key terms: multi-teacher on-policy distillation, MOPD, teacher, student, prompt-dependent mixture weight, reverse-KL loss, reverse KL, advantage, GRPO, group-level normalization
12.4 AI feedback
This section introduces AI feedback as a way to generate or augment quality judgments, places RLAIF and Constitutional AI in that context, and identifies uses beyond preference labels.
- AI feedback and RLAIF scope: Distinguish AI feedback from RLAIF and describe where AI feedback fits in the RLHF pipeline. Explain that AI feedback can augment or replace human feedback, particularly for preference data collection and reward model training.
Why it matters: This distinction makes clear that AI feedback is broader than one training recipe. - Constitutional AI historical relationship: Explain the relationship between Constitutional AI and RLAIF as presented here.
Why it matters: It prevents a prominent example of AI feedback from being mistaken for the whole category. - Uses beyond preference labels: Describe the additional uses of AI-judgment infrastructure identified in this section, without detailing how they work.
Why it matters: It shows why AI-generated judgments matter beyond collecting training preferences.
Key terms: AI feedback, RL from AI feedback (RLAIF), preference data collection, reward model training, Constitutional AI (CAI), RLAIF, LLM-as-a-judge, scalable evaluation, rubric-based rewards, verifiable answers
12.4.1 Balancing AI and human feedback data
This section explains why AI feedback is attractive for RLHF, why it has not replaced human feedback, and how to think about balancing the two.
- AI feedback cost advantage: Explain why AI feedback makes large-scale RLHF experimentation and evaluation more practical than human feedback.
Why it matters: Cost helps explain why AI feedback is useful even when it is not the best source of every judgment. - Task-dependent feedback balance: Explain why the section does not recommend replacing all human feedback with AI feedback, and describe its practical advice for starting and scaling a pipeline.
Why it matters: A cheaper source of feedback is not necessarily a complete substitute across every task. - Noise and bias tradeoff: Distinguish the section’s rules of thumb for human versus synthetic preference data and explain the practical consequence of each.
Why it matters: The noise–bias distinction explains why easy-to-obtain AI labels may still need to be balanced with human data.
Key terms: AI feedback, human preference data, evaluation, human feedback, RLAIF, RLHF, human data, synthetic preference data, high-noise, low-bias
12.4.2 Building specific LLMs for judgment
This section explains why LLM judgments can be biased and why a separately trained judge is usually unnecessary despite that concern.
- LLM judge reliability limits: Explain why using an LLM to judge responses raises calibration concerns, including inconsistent evaluations and self-preference bias.
Why it matters: These biases matter when model judgments are used to label training data. - Choosing a judgment approach: Describe the alternatives mentioned for improving or specializing model judgments, and explain when the section says training your own judge may be warranted.
Why it matters: It helps you weigh judge bias against the cost and likely value of building a separate labeling model.
Key terms: LLM-as-a-judge, calibration, self-preference bias, critic models, scaling inference, self-refinement, tournament ranking
12.5 Constitutional AI
This section explains how Constitutional AI uses human-written principles to generate revised instruction examples and AI-labeled preference data for RLHF.
- Principle-guided instruction data: Trace how Constitutional AI creates instruction data by repeatedly critiquing and revising a model response against sampled principles, then fine-tuning on the final revision.
Why it matters: It shows how written principles can shape synthetic examples used to fine-tune a model. - Principle-guided preference data: Describe how a feedback model uses constitutional principles to choose between two completions, producing synthetic preference data that can be used in RLHF.
Why it matters: It explains why this approach is called reinforcement learning from AI feedback: a model supplies the preference judgment.
Key terms: Constitutional AI, constitution, principles, critique, instruction data, preference data, feedback model, RLAIF, generative reward model
12.5.1 Further reading on CAI
Surveys extensions of Constitutional AI that use written principles in model alignment and feedback, while noting that few are documented as clear improvements to RLHF recipes.
- Principle Based Alignment Extensions: Describe how related work uses written policies or principles to guide models, reward models, or human judgments, and explain the section’s caution about their status in RLHF recipes.
Why it matters: It helps distinguish promising uses of principles from established improvements to a training recipe.
Key terms: CAI, Model Spec, deliberative alignment, principle-driven preferences, feedback
12.6 Rubrics: Prompt-specific AI feedback for training
The section explains how prompt-specific rubrics turn AI feedback into rewards for training on tasks without clearly verifiable answers, and how those rubrics can be generated at scale.
- Rubrics as training rewards: Explain why rubrics are useful for prompts without clearly verifiable answers and how they support reinforcement learning.
Why it matters: Rubrics extend reward-based training to answers that cannot simply be marked right or wrong. - Prompt-specific weighted criteria: Describe what the example rubrics ask a grader to check and how they express the importance of different criteria.
Why it matters: The criteria tell a grader what to look for, while priority markings distinguish crucial requirements from lesser qualities or pitfalls. - Scalable rubric generation: Explain the cost of generating a rubric for every training prompt and the section's approach to reducing that burden.
Why it matters: The approach retains prompt-specific feedback without requiring every rubric to be built wholly from scratch.
Key terms: rubrics, nearly verifiable criteria, RL, rewards, Hard Rule, Principle, Essential, Important, Optional, Pitfall
12.7 Summary
Where this is used later
- (chapter intro) (chapter 16) · Returns to evaluation as an application area.
- Character training (chapter 17) · Constitutional AI as a synthetic-data approach used in post-training.
- Model specifications (chapter 17) · The earlier discussion of Constitutional AI provides context for the comparison with guiding principles used in training.
Concepts explained in this chapter
- distillationDistillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.
- AI feedbackAI feedback is evaluative information about model outputs or actions generated by an artificial-intelligence system rather than directly supplied by people.
- synthetic preference dataSynthetic preference data is preference data generated by models or algorithmic processes rather than directly collected from human judgment.
- RLAIFReinforcement learning from AI feedback is a reinforcement-learning alignment technique that uses AI-generated evaluative feedback as its reward or preference signal.
- synthetic dataSynthetic data is data generated by an algorithmic or model-based process rather than directly collected from the target phenomenon.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- evaluatorLLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.
- off-policy distillationOffline knowledge distillation trains a student on data or outputs generated in advance by a teacher rather than by the current student policy.
- instruction dataAn instruction dataset is a collection of prompts or instructions paired with target responses for supervised model adaptation.
- rubricA rubric is a structured set of weighted evaluation criteria for judging the quality of a response to a specific task or prompt.
- on-policy distillationOn-policy distillation is distillation in which training examples or trajectories are sampled from the current student policy and evaluated against a teacher policy.
- Constitutional AIConstitutional AI is a model-training approach that uses a set of principles to guide and evaluate model behavior.
- reverse KL divergenceReverse KL divergence is the Kullback-Leibler divergence from a target distribution to a reference distribution, penalizing probability mass assigned by the target where the reference assigns little probability.
- constitution-as-code documentA constitution is a human-written set of principles specifying aspects of model behavior to assess during critique.
- post-trainingPost-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
- principleA principle is a rubric criterion expressing a desirable quality that guides evaluation without being an absolute requirement.
- Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- evaluationModel evaluation is the systematic assessment of a model's behavior or performance against defined tasks, criteria, or benchmarks.
- cross entropy lossCross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
- model collapseModel collapse is progressive degradation of a model's effective training distribution and generalization caused by repeated training on model-generated data.
- advantage functionAn advantage function estimates how much better an action is than the policy's expected value at a state.
- data driftDistribution mismatch is a discrepancy between the probability distribution that generated training data and the distribution of the policy being optimized.
- generative reward modelA generative reward model is a reward model that generates evaluative judgments or preference assessments rather than only emitting a scalar score.
- verifiable taskA verifiable domain is a problem area in which candidate outputs can be objectively checked for correctness.
- OpenAI Model SpecA model specification is a written behavioral specification defining intended behaviors and principles for a language model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.