Policy gradient method

As explained in Build a Reasoning Model (From Scratch)

The reinforcement-learning objective used to adjust a policy so actions with better outcomes become more likely and worse ones less likely.

Policy gradient loss combines rollout advantages with log probabilities to push the policy toward actions that earned better outcomes and away from worse ones. section 6.2.1 section 6.9

It is computed as the negative mean of advantage-weighted sequence log probabilities, with advantages detached so only policy parameters receive gradients. section 6.9

The update flow uses it to:

• turn rollout logprobs into a tensor before scoring them. section 6.9
• apply a sign flip so a maximization objective fits a minimization optimizer. section 6.9
• feed the result into the later model weight update step. section 6.2.1 section 6.9

Concrete loss computation

A listed implementation stacks rollout log probabilities and computes pg_loss = -(advantages.detach() * logps).mean(). section 6.9

Figure 6.8 Overview of the GRPO algorithm for RLVR using a chef analogy. Multiple rollouts are generated and scored, relative advantages are computed, and a policy gradient objective with a KL-based regularization loss term is used to update the model parameters.

Shows the GRPO flow where multiple rollouts are scored, relative advantages are computed, and the policy gradient objective updates parameters. section 6.2.1

Figure 6.17 The fifth stage in the GRPO pipeline computes the policy gradient loss that we’ll use to update the model. Stage 6, the model weight update, will be implemented later as part of the training loop.

Places policy gradient loss as stage five in the GRPO pipeline, before the separate weight update stage. section 6.9

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Defines the loss, shows the sign flip, and explains detach; it stops short of the later weight update implementation.

Examples

Uses a chef analogy, a worked sign-flip example, and a PyTorch listing with stacked logps.

What it assumes

Assumes rollout advantages and log probabilities are already available, and introduces the loss computation from there.

What you can do after

Apply it via PyTorch to compute a minimizable loss that increases high-advantage rollout likelihoods.

How it connects

Links advantages, log probabilities, policy updates, KL-based regularization, and model weight update.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackPolicy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA policy gradient is a reinforcement-learning method that optimizes a policy by estimating gradients of expected return with respect to policy parameters.

Compare how 3 books explain policy gradient method →

Related concepts

• Proximal Policy Optimizationincludes · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• log probabilityuses · A log probability is the logarithm, usually the natural logarithm, of a probability assigned to an event, outcome, token, or sequence.
• REINFORCEincludes · REINFORCE is a Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of selected actions’ log probabilities.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
• advantage functionuses · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.
• reference modeluses · A reference model is a fixed policy distribution that serves as a baseline for comparing or constraining changes to an optimized policy.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.