liveBook · Manning

Best-of-N

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Best-of-N reranking is a selection method that scores multiple candidate solutions and retains the highest-scoring candidate.

Best-of-N (BoN) uses a trained reward model during inference: sample N candidate responses from the policy at temperature greater than zero, score every candidate, and return the highest-scoring response. section 7.5.7

Apply BoN in three steps:

Choosing the sample budget

BoN@4 can close a noticeable portion of the gap between a base model and a full RLHF-trained model; BoN@64 can rival PPO outputs on some narrow preference benchmarks. Larger N costs proportionally more decoding and can increase reward-model overoptimization. section 7.5.7

Figure 7.9 MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.

Compare process-reward-model, outcome-reward-model, and majority-vote reranking as candidate-sample counts increase; process supervision remains the strongest filter in the reported MATH comparison. section 7.7.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains sampling, scoring, selection, linear compute cost, KL drift, and reward-model calibration risk.
Examples
Uses BoN@4 and BoN@64 comparisons, plus MATH reranking with process and outcome reward models.
What it assumes
Assumes a policy capable of sampling multiple responses and a trained reward model; teaches the selection procedure directly.
What you can do after
Apply it via a trained reward model by sampling, scoring, and selecting candidates under a chosen N budget.
How it connects
Connects reward models with PPO, KL penalties, RLHF, reward hacking, process supervision, and outcome reward models.

Also explained in

Compare how 2 books explain best-of-N →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.