Clipped policy ratio

As explained in Build a Reasoning Model (From Scratch)

A policy-optimization constraint that limits how far a new policy’s probability ratio can move from an old policy’s ratio for the same sampled response.

A clipped policy ratio is a GRPO update constraint that limits how far the current policy’s probability ratio can move from a previous policy for the same sampled response. section 7.4 section 7.4.1

It compares old and new sequence logprobs for the same prompt and response, then clamps the ratio so a single update cannot change behavior too abruptly. section 7.4.1 section 7.5

The implementation uses this pattern:

• Copy the current model as a reference policy and sample fixed rollouts from it. section 7.4.2
• Compute old_logps from the reference policy and new_logps from the updated model on the same responses. section 7.4.1
• Form ratio from the logprob difference, clamp it to a bounded range, and use the minimum of unclipped and clipped objectives in the loss. section 7.4.1 section 7.4.1

Training workflow

One described workflow keeps a large rollout pool fixed, processes it in minibatches, then refreshes the reference model. A smaller approximation repeats the same logic on fresh batches during each step. section 7.4.2

Figure 7.10 We have plotted basic and advanced GRPO training metrics. Now we’ll modify the GRPO algorithm to add clipped policy ratios.

Figure 7.10 situates clipped policy ratios as the next GRPO modification after basic metrics analysis. section 7.4

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Explains the ratio calculation, clamping step, and clipped loss, but skips the mathematical derivation.

Examples

Uses old and new logprobs, fixed rollouts, minibatch updates, and a metrics run to illustrate the method.

What it assumes

Assumes familiarity with policy gradients and logprobs, while defining the ratio from scratch.

What you can do after

Apply it in GRPO training to limit update size and reduce instability.

How it connects

Relates to GRPO, policy ratio, policy gradient loss, and KL divergence.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

• Group Relative Policy Optimizationpart of · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
• log probabilityuses · A log probability is the logarithm, usually the natural logarithm, of a probability assigned to an event, outcome, token, or sequence.
• policyuses · A policy is a rule, strategy, or mapping that governs or determines decisions, actions, permissions, or behavior under specified conditions.
• gradient descentaddresses · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• importance weightcontrasts with · An importance weight is the ratio of an outcome’s probability under a target distribution or policy to its probability under a sampling or reference distribution or policy, used to reweight that outcome.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.