liveBook · Manning

Rejection sampling

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Rejection sampling is a filtering method that retains generated candidates meeting a specified acceptance criterion and discards the others.

Rejection sampling filters training data by generating multiple candidates, scoring them with a reward model, and retaining only candidates whose reward exceeds a chosen threshold. The policy is then supervised-fine-tuned on the retained responses. section 7.5.7

A typical rejection-sampling round proceeds as follows:

  • Generate several responses for each prompt from the current policy. section 7.5.7
  • Score every response with the reward model. section 7.5.7
  • Keep responses whose scores pass the acceptance threshold and discard the rest. section 7.5.7
  • Supervised-fine-tune the policy on the filtered corpus, then optionally repeat the process with the improved policy. section 7.5.7

Rejection sampling fine-tuning, also called RSFT or BoN-SFT, is simpler than PPO: it uses filtered supervised learning rather than on-policy sampling, KL regularization, or value-function machinery. Its losses are described as numerically stable compared with sometimes-explosive PPO gradients. section 7.5.7

Iterated filtering

A post-training pipeline can train on one rejection-sampled dataset, generate fresh candidates from the improved policy, filter them again, and repeat. This approach was used before a final PPO stage in Llama 2 and can also replace PPO in repeated rounds. section 7.5.7

The central risk is reward-model miscalibration. If the model rewards a failure mode, filtering concentrates that failure in the training data, and later iterations can amplify it. Filtering therefore trades PPO complexity for dependence on reward quality. section 7.5.7

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains candidate generation, reward scoring, thresholding, supervised fine-tuning, iteration, and reward-model failure risks.
Examples
Uses iterated post-training rounds and Llama 2’s rejection-sampling stage before final PPO as concrete examples.
What it assumes
Assumes a candidate-generating policy and reward model, while defining the filtering and fine-tuning loop.
What you can do after
Apply it via supervised fine-tuning to build filtered training corpora, iterate policy improvement, and inspect reward-model failure modes.
How it connects
Connects rejection sampling to reward models, supervised fine-tuning, PPO, KL regularization, and best-of-N sampling.

Also explained in

Compare how 5 books explain rejection sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.