9 Rejection sampling
This chapter covers
- What rejection sampling is
- Why rejection sampling may be used instead of RL
Rejection sampling (RS) is one of the most widely used but least documented methods in preference fine-tuning. Many prominent RLHF papers use it as a core component of their training pipeline, yet no canonical implementation or explanation of why it works so well exists. RS can be applied at multiple points in the training pipeline—after instruction fine-tuning, after RL-based optimization, or even after RLVR—making it a versatile but hard-to-place tool. Combined with its underdocumented nature, this is why it appears here at the end of the core optimization methods. RS operates by curating new candidate completions, filtering them based on a trained reward model (RM), and then fine-tuning the original model only on the top completions (the same loss function as instruction tuning).
The name originates from computational statistics [1], where we wish to sample from a complex distribution but do not have a direct method to do so. To alleviate this, we sample from a distribution that is simpler to model and use a heuristic to check whether the sample is permissible. With language models, the target distribution is high-quality completions to prompts, the filter is an RM, and the sampling distribution is the current model.