Synthetic data is training data generated by a program rather than collected and labeled from the real world. It sounds like an easy shortcut, but the generator must already know what a correct example looks like. section 1.27
When simulation works
AlphaZero generated millions of Go games by playing against itself. Simulation worked because the program could calculate the outcome of each game: it could tell who won and create labeled training examples. section 1.27
The same shortcut usually fails for open-ended tasks. A program generating LLM training examples would need to predict the correct next word; a program labeling images would need to recognize their categories. In both cases, the supposed data generator already requires the capability being trained. Synthetic data can therefore help in narrow, easily scored environments, but it is not a general replacement for collecting and labeling real data. section 1.27
