Best-of-N (BoN) uses a trained reward model during inference: sample N candidate responses from the policy at temperature greater than zero, score every candidate, and return the highest-scoring response. section 7.5.7
Apply BoN in three steps:
- Sample N responses from the policy. section 7.5.7
- Score each response with the reward model. section 7.5.7
- Return the candidate with the highest score. section 7.5.7
Choosing the sample budget
BoN@4 can close a noticeable portion of the gap between a base model and a full RLHF-trained model; BoN@64 can rival PPO outputs on some narrow preference benchmarks. Larger N costs proportionally more decoding and can increase reward-model overoptimization. section 7.5.7
Figure 7.9 MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.
Compare process-reward-model, outcome-reward-model, and majority-vote reranking as candidate-sample counts increase; process supervision remains the strongest filter in the reported MATH comparison. section 7.7.2

