A gradient is a vector of derivatives indicating how a scalar function changes with respect to its parameters. In model optimization, it supplies the direction and magnitude for changing parameters to improve an objective. In RLHF, policy-gradient methods use gradients of expected reward to translate reward-model scores into policy behavior. section 6.6 section 8.1.2
Gradient estimates take different forms depending on the optimization setting:
- REINFORCE uses Monte Carlo returns with a batch-level baseline, while RLOO uses leave-one-out comparisons across multiple completions; neither requires a learned value function. section 6.6
- PPO estimates per-token advantages with a learned value function and GAE, whereas GRPO normalizes advantages relative to groups of completions. section 6.6
- DPO differentiates a sigmoid preference objective built from chosen and rejected completion log-probability ratios relative to a reference policy. section 8.1.2
- Each token can receive a different gradient magnitude and direction according to its contribution to the preference margin. section 8.4
From preference signal to update
A policy generates several completions, a reward model scores them, and a policy-gradient estimate updates the model toward higher-reward behavior. Baselines and advantage functions reduce estimate variance; clipping, aggregation, and truncated-generation handling affect stability. section 6.6
Equation 8.5
\[\label{eq:rlhf_opt_eq_repeat} \max_{\pi} \mathbb{E}_{x \sim \mathcal{D}}\mathbb{E}_{y \sim \pi(y|x)} \left[r_\theta(x, y)\right] - \beta \mathcal{D}_{\text{KL}}\left(\pi(y|x) \| \pi_{\text{ref}}(y|x)\right).\]The RLHF objective shows the scalar target whose gradient balances expected reward against divergence from a reference policy. section 6.6
