Rejection sampling

As explained in Reinforcement Learning from Human Feedback

Rejection sampling is a sampling method that accepts or filters candidate outputs according to a specified criterion or scoring procedure.

Rejection sampling (RS) is a preference-fine-tuning method that generates candidate completions, scores them with a trained reward model, and trains on only the strongest outputs. It can follow instruction fine-tuning, reinforcement-learning optimization, or RLVR. The current model supplies samples, while the reward model acts as a filter for high-quality completions. section 9.0 section 9.4

A practical rejection-sampling pipeline consists of four stages:

• Select training prompts and a reward model; reusing every instruction-tuning prompt is simple but may overfit. section 9.1
• Generate multiple completions from the starting checkpoint, configuring temperature, top-p, sequence length, and completions per prompt. section 9.1
• Rank completions with the reward model and optionally deduplicate them; retain either the best completion per prompt or the globally highest-scoring K. section 9.1 section 9.4
• Instruction-fine-tune the starting checkpoint on the retained completions. section 9.1

Rejection sampling versus Best-of-N

Both methods generate and reward-score several completions. Rejection sampling fine-tunes the model on selected outputs; Best-of-N uses the highest-scoring output only at inference time and does not update the model. section 9.4

Figure 9.1 RS overview

Use the pipeline overview to inspect how prompt selection, generation, reward ranking, and supervised fine-tuning connect. section 9.1

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Explains four pipeline stages and selection strategies, but defers prompt, reward-model, sequencing, and deduplication choices to experimentation.

Examples

Uses WebGPT, Helpful and Harmless, process reward models, Llama 2 Chat, and DeepSeek R1 as pipeline examples.

What it assumes

Assumes a starting checkpoint, selected prompts, and a trained reward model; teaches the generate-rank-fine-tune loop.

What you can do after

Operate/configure a generate-rank-fine-tune pipeline and choose between per-prompt and global completion selection.

How it connects

Connects reward modeling, instruction fine-tuning, reinforcement learning, preference learning, and Best-of-N sampling.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Fabulous Adventures in Data Structures and AlgorithmsThe rejection algorithm samples candidate outcomes and discards candidates that fail a probabilistic acceptance test.
• 

  liveBookmanning.com
  Building LLM Applications with DSPyRejection sampling is a method that accepts candidate samples meeting a criterion and rejects those that do not.

Compare how 3 books explain rejection sampling →

Related concepts

• reward modeluses · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
• reinforcement learningalternative to · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• data curationkind of · Data curation is the systematic selection, review, cleaning, organization, and maintenance of data for a particular purpose.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.