Rejection sampling filters training data by generating multiple candidates, scoring them with a reward model, and retaining only candidates whose reward exceeds a chosen threshold. The policy is then supervised-fine-tuned on the retained responses. section 7.5.7
A typical rejection-sampling round proceeds as follows:
- Generate several responses for each prompt from the current policy. section 7.5.7
- Score every response with the reward model. section 7.5.7
- Keep responses whose scores pass the acceptance threshold and discard the rest. section 7.5.7
- Supervised-fine-tune the policy on the filtered corpus, then optionally repeat the process with the improved policy. section 7.5.7
Rejection sampling fine-tuning, also called RSFT or BoN-SFT, is simpler than PPO: it uses filtered supervised learning rather than on-policy sampling, KL regularization, or value-function machinery. Its losses are described as numerically stable compared with sometimes-explosive PPO gradients. section 7.5.7
Iterated filtering
A post-training pipeline can train on one rejection-sampled dataset, generate fresh candidates from the improved policy, filter them again, and repeat. This approach was used before a final PPO stage in Llama 2 and can also replace PPO in repeated rounds. section 7.5.7
The central risk is reward-model miscalibration. If the model rewards a failure mode, filtering concentrates that failure in the training data, and later iterations can amplify it. Filtering therefore trades PPO complexity for dependence on reward quality. section 7.5.7




