Policy gradient method

As explained in Reinforcement Learning from Human Feedback

Policy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.

Policy gradient methods optimize a parameterized policy directly from sampled outcomes. In RLHF, a language model generates completions for prompts, a reward model scores them, and gradient updates make higher-scoring behavior more likely. A frozen reference policy commonly contributes a KL penalty, limiting drift while preference feedback is translated into model behavior. section 6.0 section 3.3

The central update has the form Δθ ∝ Ψt ∇θ log πθ(at|st). The log-probability gradient identifies parameters that made an action likely; Ψt scores its quality. Positive scores reinforce an action, negative scores discourage it. For language models, states and actions can be read as prompts and completions, or as token-level sequential decisions. section 6.2 section 6.3.1

Figure 6.1 Overview of the RLHF training loop. A prompt from the dataset is passed to the tuned policy, which generates a completion. The reward model scores this completion, and the frozen initial model (typically the instruction-tuned model before RL) computes log probabilities on the same text to calculate a Kullback–Leibler penalty that prevents excessive drift. The combined reward signal then drives an RL update to the policy parameters.

Follow the operational loop: sample a completion from the current policy, score it, subtract the reference-policy KL cost, then update the policy from the combined reward. section 6.0

The objective is expected trajectory reward, J(θ) = Eτ∼πθ[R(τ)]. The likelihood-ratio derivation turns its gradient into sampled policy log-probabilities weighted by return: ∇θJ(θ) = E[Σt ∇θlogπθ(at|st) R(τ)]. Environment dynamics and the initial-state distribution disappear from the parameter gradient because they do not depend on θ. Monte Carlo rollouts approximate the expectation in a batch. section 6.2.1

Raw-return updates are noisy, especially with small rollout sets, stochastic generation, or sparse rewards. A baseline reduces variance without changing the expected gradient. The common advantage signal A(s,a) = Q(s,a) − V(s) instead reinforces actions better than expected and suppresses those worse than expected. REINFORCE uses Monte Carlo returns and may use a batch or state-dependent baseline; it does not require a learned value network. section 6.2.2 section 6.2.3

Practical RLHF policy-gradient variants change how the score signal and update are stabilized:

• REINFORCE and RLOO use Monte Carlo returns; RLOO forms a leave-one-out baseline across several completions per prompt. section 6.6
• PPO uses importance ratios, clipping, and a learned value function to constrain a policy change; its per-token advantages can use GAE. section 6.6 section 6.3.5
• GRPO replaces PPO's learned value function with group-relative advantage normalization across multiple completions, and is popular for reasoning tasks. section 6.6

Advantage-weighted loss

For each sampled completion, sum token log probabilities only over completion tokens, multiply by its advantage, negate, average, and backpropagate. With positive advantage, minimizing the loss raises the completion probability; with negative advantage, it lowers it. PPO instead weights the advantage by a new-to-old probability ratio and clips that ratio to prevent overly large changes. section 6.2.1 section 6.3.1 section 6.3.5

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Derives the likelihood-ratio gradient, implements masked losses, and compares variance reduction and PPO-style stabilization.

Examples

Uses prompt completions, reward-model scoring, masked token log-probabilities, Monte Carlo rollouts, and clipped PPO probability ratios.

What it assumes

Assumes reinforcement-learning notation for states, actions, trajectories, returns, and parameterized policies.

What you can do after

Apply it via RLHF losses that reinforce advantageous completions while controlling variance and policy drift.

How it connects

Connects reward models, KL divergence, advantage functions, value functions, REINFORCE, PPO, GRPO, and importance sampling.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA policy gradient is a reinforcement-learning method that optimizes a policy by estimating gradients of expected return with respect to policy parameters.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)The reinforcement-learning objective used to adjust a policy so actions with better outcomes become more likely and worse ones less likely.

Compare how 3 books explain policy gradient method →

Related concepts

• Proximal Policy Optimizationincludes · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• log probabilityuses · A log probability is the logarithm, usually the natural logarithm, of a probability assigned to an event, outcome, token, or sequence.
• REINFORCEincludes · REINFORCE is a Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of selected actions’ log probabilities.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
• advantage functionuses · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.
• reference modeluses · A reference model is a fixed policy distribution that serves as a baseline for comparing or constraining changes to an optimized policy.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.