Synthetic data is training data produced by an algorithmic or model-based process rather than collected directly from the target phenomenon. In language-model post-training, it expands the supply of prompts, completions, critiques, preference labels, and quality judgments. Its practical value is lower-cost, faster iteration than relying exclusively on human feedback, allowing data pipelines to shape model behavior at scale. section 12.0 section 12.1
A typical pipeline starts from seed examples or existing prompts, then uses capable language models to generate or modify prompts and produce candidate completions. Models can also score, filter, or compare those completions, turning generated outputs into instruction or preference data for fine-tuning and RLHF. This makes synthetic data a family of pipeline operations, not merely a generated dataset: output quality, teacher diversity, and filtering determine whether added scale is useful. section 12.1 section 4.2
Synthetic data can fill several distinct post-training roles:
- Generate new prompts from seed examples or rewrite existing prompts to broaden training coverage. section 12.1
- Generate completions that become the targets learned during instruction tuning; prompt distributions should resemble downstream tasks. section 12.1 section 4.2
- Produce AI feedback, scores, or pairwise comparisons that form preference data for RLHF or RLAIF. section 12.1 section 12.5
- Filter and deduplicate candidate outputs before training, limiting repetitive or low-quality examples. section 12.1
The main failure mode is model collapse: repeatedly learning from narrow, unfiltered outputs of a single model can reduce diversity, underrepresent rare facts or styles, amplify small mistakes, and weaken generalization. This is not presented as an inevitable consequence of synthetic data. Mixing real or human data, using diverse teachers, deduplicating, and applying strong quality filters can avoid the collapse regime. section 12.1
Constitutional AI: principles become synthetic supervision
A human-written constitution supplies principles such as avoiding violence and being truthful. For instruction data, a model repeatedly critiques and revises a completion against sampled principles; the final revision becomes a prompt–completion training example. For preference data, a feedback model receives a prompt, principles, and two completions, then selects the better completion. That synthetic preference signal supports RLHF under the name RLAIF. section 12.5
Tülu 3: a multistage synthetic-data recipe
Tülu 3 begins with roughly one million primarily synthetic instruction-tuning examples drawn from frontier models, establishing general instruction following and foundations for mathematics and coding. It then uses roughly one million on-policy preference pairs and a smaller RLVR run on about 10,000 prompts. The sequence illustrates that synthetic data can be one stage in a broader post-training system rather than the entire training process. section 3.2.2
Figure 3.7 A summary of the Tülu 3 recipe with target skills and multistep training recipe. Lambert et al. 2024, license CC-BY.
This recipe diagram makes the ordering and relative scale of synthetic instruction tuning, preference training, and RLVR concrete. section 3.2.2
