12 Synthetic data
This chapter covers
- Defining synthetic data
- Why synthetic data actually works
- Distilling models from other models
- Teacher–student knowledge distillation methods
- Constitutional AI and methods of
AI feedback - Learning from rubrics and future directions of synthetic data
Reinforcement learning from human feedback is deeply rooted in the idea of maintaining human influence in the models we are building. When the first models were trained successfully with RLHF, human data was the only viable way to improve the models by creating high-quality responses to questions that provided reliable, specific feedback data.
As AI models got better, this assumption rapidly broke down. The possibility of synthetic data, which is far cheaper and easier to iterate on, enabled the proliferation of RLHF by lowering the price of experiments and research. This translated into RLHF being the early center of attention in the broader “post-training” shaping models. This chapter provides a cursory overview of how and why synthetic data is replacing or expanding many pieces of the RLHF pipeline.