liveBook · Manning

Monte carlo reward estimate

As explained in Reinforcement Learning from Human Feedback

A Monte Carlo return estimate computes a return from rewards observed along complete sampled trajectories rather than bootstrapping from a value estimate.

A Monte Carlo reward estimate computes a return from rewards observed along a complete sampled trajectory. Unlike bootstrapping, it does not require a learned value function. section 6.2.11 section 6.6

In policy-gradient training, the estimate functions as follows:

  • REINFORCE uses Monte Carlo returns with a state-based or batch-level baseline to reduce gradient variance. section 6.2.11 section 6.6
  • RLOO retains the Monte Carlo approach but replaces the baseline with the average reward of other completions for the same prompt. section 6.2.11
  • PPO instead estimates per-token advantages with a learned value function and generalized advantage estimation. section 6.6

REINFORCE update

For each sampled completion, REINFORCE weights log-probability terms by the Monte Carlo return minus a baseline, then optimizes the resulting policy-gradient loss. section 6.2.11

Table 6.1 Comparing policy gradient algorithms. IS: Importance Sampling.
Method IS granularity Clipping style Advantage
REINFORCE None None Monte Carlo baseline
RLOO None None Leave-one-out
PPO Token Objective (bilateral) Learned value function
GRPO Token Objective (bilateral) Group-relative
GSPO Sequence Objective (bilateral) Group-relative
CISPO Token Weights (stop-grad) Group-relative

Compare Monte Carlo baselines with learned-value and group-relative advantage methods across policy-gradient algorithms. section 6.2.11

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Contrasts Monte Carlo returns with learned value-function and group-relative estimates, but does not derive return computation in detail.
Examples
Uses REINFORCE's baseline-subtracted return and RLOO's leave-one-out reward average as concrete estimation patterns.
What you can do after
Apply it via REINFORCE or RLOO to weight policy updates without implementing a learned value function.
How it connects
Connects Monte Carlo returns to policy gradients, REINFORCE, RLOO, baselines, advantage functions, and value functions.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.