liveBook · Manning

RLOO

As explained in Reinforcement Learning from Human Feedback

RLOO, or REINFORCE leave one out, is a policy-gradient algorithm that estimates each sample's advantage relative to the average reward of the other samples for the same prompt.

RLOO (REINFORCE leave one out) is a policy-gradient method for training a language-model policy from rewards on multiple completions of the same prompt. Rather than asking whether a completion is good in absolute terms, it asks whether it scored better than its sibling completions. This relative baseline reduces gradient variance without learning a value function. section 6.2.4 section 6.6

For a prompt s with K sampled completions a₁…aₖ and rewards R(s,a₁)…R(s,aₖ), completion k receives the leave-one-out baseline b(s,aₖ) = (1/(K−1)) Σᵢ≠ₖ R(s,aᵢ). Its advantage is A(s,aₖ) = R(s,aₖ) − b(s,aₖ). Excluding the completion’s own reward makes the baseline independent of the evaluated action, preserving an exactly unbiased gradient estimator. section 6.2.4

Figure 6.3 RLOO architecture. Multiple completions per prompt provide a leave-one-out baseline for advantage estimation without learning a value function.
figure

Multiple completions from one prompt supply the comparison set: each completion is rewarded relative to the mean reward of all the others. section 6.2.4

The policy loss applies each sequence-level advantage to the log probabilities of that completion’s tokens. A positive advantage raises the probability of every sampled token; a negative advantage lowers it. Because RLOO has no critic, every token in a completion receives the same final sequence-level signal, unlike PPO’s token-specific value-based advantages. section 6.2.11 section 6.2.4

Vectorized per-prompt advantages

Store total rewards for N prompts with k completions as a flat tensor, reshape it to (k, N), then compute baseline = (rewards.sum(0) − rewards) / (k − 1). Subtracting that baseline yields a (k, N) advantage tensor, which can be flattened to align again with the completions. This implementation ensures every baseline compares only completions answering the same prompt. section 6.3.7

RLOO’s practical trade-offs are:

  • It requires multiple trajectories per prompt; a single completion cannot form a leave-one-out baseline. section 6.2.4
  • It avoids a learned value network and importance-sampling ratio, making it simpler than PPO but limiting the extra update machinery those components support. section 6.2.11
  • It is closely related to group-relative methods: Dr. GRPO’s unnormalized group-mean advantage is equivalent to RLOO up to a constant scaling factor. section 6.2.8
  • KL regularization can be folded into RLOO rewards, whereas canonical GRPO applies its KL penalty directly in the loss; PPO-style clipping can also be combined with RLOO advantages. section 6.2.4
Table 6.1 Comparing policy gradient algorithms. IS: Importance Sampling.
Method IS granularity Clipping style Advantage
REINFORCE None None Monte Carlo baseline
RLOO None None Leave-one-out
PPO Token Objective (bilateral) Learned value function
GRPO Token Objective (bilateral) Group-relative
GSPO Sequence Objective (bilateral) Group-relative
CISPO Token Weights (stop-grad) Group-relative

Use the comparison table to distinguish RLOO’s leave-one-out advantage and lack of importance sampling from REINFORCE, PPO, GRPO, GSPO, and CISPO. section 6.2.11

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Derives the baseline and advantage, explains unbiasedness and token assignment, and shows tensor reshaping without a full training loop.
Examples
Demonstrates reshaping rewards into a completions-by-prompts tensor and computing leave-one-out baselines before flattening advantages.
What it assumes
Assumes familiarity with rewards, policy gradients, prompts, completions, and token log probabilities; introduces the leave-one-out estimator explicitly.
What you can do after
Apply it via reward tensors by generating grouped completions, computing leave-one-out advantages, and weighting completion token log probabilities.
How it connects
Connects REINFORCE, PPO, GRPO, KL regularization, importance sampling, learned value functions, and policy gradients.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.