Chapter 6: Reinforcement learning

Introduction

Introduces the RLHF training loop in which a policy generates completions, a reward model scores them, and reinforcement learning repeatedly updates the policy.

• RLHF policy update loop: Trace how the policy generates completions for training prompts, the reward model scores them, and the reinforcement learning optimizer uses that feedback to update the policy over repeated batches.
  Why it matters: This loop connects reward-model feedback to changes in the language model being trained.

Key terms: policy, reward model, on-policy data, gradient updates

6.1 The role of reinforcement learning in RLHF

This section places RLHF’s reinforcement learning stage in the policy-gradient family, identifies the models it uses, and explains the main ways its algorithms differ.

• Policy-gradient approach to RLHF: Distinguish policy-gradient methods from value-based methods and explain why the RLHF algorithms discussed here belong to the policy-gradient family.
  Why it matters: This distinction explains why the chapter focuses on updates to the language model’s policy.
• Models in the RL stage: Describe the roles of the policy, reward model, frozen reference model, and PPO’s additional value function.
  Why it matters: Keeping these model roles separate makes the later algorithm descriptions easier to follow.
• Differences among RLHF algorithms: Explain the main dimensions on which the section says RLHF algorithms differ, and describe the stated appeal of REINFORCE-style methods relative to PPO.
  Why it matters: These are the comparison points to listen for when the later algorithms are introduced.

Key terms: value-based methods, value functions, policy-gradient methods, policy, recently generated samples, reward model, reference model, Kullback–Leibler (KL) penalty, value function, PPO

6.2 Policy gradient algorithms

This section introduces the policy-gradient update, the expected-return objective it serves, and how RLHF estimates that objective from language-model rollouts.

• Policy gradient update: Explain how the policy-gradient update combines a direction that makes an action more likely with a scalar score of how good the outcome was, and how the score’s sign affects the action’s future likelihood.
  Why it matters: This is the basic update that the chapter’s policy-gradient algorithms build on.
• Return and discounting: Describe return as discounted future rewards and explain what the discount factor changes, including the choice typically made for language-model RLHF.
  Why it matters: Return specifies what the policy is being optimized to increase.
• Expected return from rollouts: Explain that policy-gradient algorithms maximize expected return by weighting state values according to how often the policy visits those states, and describe how rollouts provide a practical estimate in RLHF.
  Why it matters: Rollouts connect the expected-return objective to data that can actually be sampled during training.

Key terms: policy-gradient update, policy, action, Ψ_t, return, discount factor, γ, collective completion, expected return, value function

6.2.1 Deriving the policy gradient

This section derives the policy gradient from trajectory probabilities, connects it to language-model log probabilities, and introduces lower-variance per-timestep reward signals.

• Trajectory probability factorization: Explain how a trajectory’s probability factors into initial-state, policy-action, and environment-transition terms, and why only the policy terms contribute to a gradient with respect to policy parameters.
  Why it matters: This identifies which parts of a sampled trajectory can change when the policy is updated.
• Expectation form of policy gradient: Trace how the log-derivative trick turns the gradient of expected trajectory reward into an expectation involving trajectory reward and a log-probability gradient. Explain why this form can be estimated from sampled rollouts.
  Why it matters: The expectation form connects the mathematical objective to estimates made from sampled trajectories.
• Language model gradient implementation: Describe how the surviving policy-probability terms appear when the policy is a language model and how their gradients are computed in practice.
  Why it matters: It links the trajectory derivation to the quantities available when training a language model.
• Per-timestep reward signals: Explain why the derivation replaces total trajectory reward with a per-timestep signal Ψₜ, and describe the kinds of signals listed.
  Why it matters: The choice of learning signal affects how noisy a policy update is.
• Advantage and TD relationships: Describe how the section relates state value, action value, advantage, and the TD residual, including the simplification stated for deterministic language-model state transitions.
  Why it matters: These relationships explain why value estimates can provide an action-level learning signal.

Key terms: trajectory probability, policy, environment transition dynamics, policy parameters, RL objective, log-derivative trick, expectation, Monte Carlo rollouts, language-model policy probabilities, tokens

6.2.2 Vanilla policy gradient

This section introduces vanilla policy gradient and explains why noisy return estimates motivate baselines and advantage-based updates.

• Vanilla policy gradient update: Describe how vanilla policy gradient optimizes the objective by differentiating it with respect to the policy parameters, using overall return in the simple version presented.
  Why it matters: This establishes the update that the section’s variance-reduction methods are meant to improve.
• Sources of gradient variance: Explain why vanilla policy-gradient updates can have high variance, including the effects of estimating returns from a small, noisy set of rollouts and of sparse rewards.
  Why it matters: It explains why a basic policy-gradient update may need a less noisy learning signal.
• Baselines and advantage estimates: Explain how baselines reduce variance without changing the expected gradient, and describe how advantage-based updates reinforce better-than-average actions and discourage worse-than-average ones.
  Why it matters: Relative value estimates make the learning signal less noisy without systematically shifting the gradient.

Key terms: vanilla policy gradient, policy parameters, overall return, high variance, return, rollouts, sparse rewards, baselines, advantage, unbiased

6.2.3 REINFORCE

This section introduces the components of the REINFORCE update and expresses it as a Monte Carlo policy-gradient update using an advantage.

• Three REINFORCE update components: Explain how the nonnegative factor, offset reinforcement, and characteristic eligibility contribute to a REINFORCE policy-gradient step.
  Why it matters: These pieces explain what each part of a REINFORCE update contributes.
• Advantage and Monte Carlo gradient: Describe how this section expresses REINFORCE using return and advantage, and state how it estimates the gradient.
  Why it matters: The advantage form connects this REINFORCE update to the policy-gradient forms used later.

Key terms: nonnegative factor, offset reinforcement, characteristic eligibility, baseline, learning rate, REINFORCE, Monte Carlo estimator, generalized return, advantage

6.2.4 REINFORCE leave one out

Explains how RLOO estimates an unbiased per-prompt advantage from multiple completions and contrasts its sequence-level updates with PPO and GRPO.

• Leave-one-out reward baseline: Explain how RLOO forms a baseline for each completion and why excluding that completion’s reward matters.
  Why it matters: The leave-one-out baseline is the defining difference between RLOO and standard REINFORCE here.
• Sequence-level token credit: Distinguish RLOO’s sequence-level advantage from the token-level values described for PPO.
  Why it matters: It clarifies what information RLOO does—and does not—use when updating the tokens of a completion.
• KL placement across algorithms: Describe where RLOO, PPO, and canonical GRPO place the KL penalty as explained here.
  Why it matters: The algorithms can use similar advantage ideas while applying regularization at different points.

Key terms: REINFORCE leave one out, RLOO, baseline, advantage, trajectories, sequence-level advantage, value network, critic, token-level credit assignment, KL penalty

6.2.5 Proximal policy optimization

This section explains how PPO reuses a batch of policy-generated data, limits overly large updates, and applies its objective to language-model tokens.

• PPO surrogate objective: Explain why PPO uses a surrogate objective rather than directly estimating the policy gradient.
  Why it matters: This connects PPO’s objective to the policy-gradient method it builds on.
• Old-policy importance ratio: Explain how the ratio of current-policy to old-policy probabilities lets PPO take multiple gradient steps on a batch collected by the old policy, and describe what the ratio indicates.
  Why it matters: The ratio explains how PPO can reuse a batch while the policy changes.
• Clipped update safeguard: Explain why PPO clips the policy ratio and state the clipping range described here.
  Why it matters: Clipping is the section’s answer to the risk introduced by repeatedly optimizing on an old batch.
• Per-token policy ratios: Describe how this section applies the PPO objective to language-model tokens and computes each token’s policy ratio.
  Why it matters: It connects the policy-probability ratio to the token log-probabilities used in language-model implementations.

Key terms: PPO surrogate objective, policy gradient, current policy, old policy, importance sampling, policy ratio, importance weight, clipping, ε, clipped surrogate objective

6.2.6 Understanding the PPO objective

This section explains how PPO’s clipped objective uses the policy ratio and the sign of the advantage to reinforce or suppress an action without continuing to push an already large change.

• PPO trust region behavior: Explain when PPO gives a normal policy-gradient update and how clipping makes the objective flat once a probability change has gone far enough in the desired direction.
  Why it matters: This explains why PPO can make useful policy-gradient updates without endlessly reinforcing an already large change.
• Positive advantage clipping: Trace what PPO does for a positive-advantage action as its new-to-old policy ratio moves below, within, or above the clipping interval.
  Why it matters: The positive-advantage case shows how PPO reinforces a beneficial action without over-reinforcing it.
• Negative advantage clipping: Trace what PPO does for a negative-advantage action as its new-to-old policy ratio moves below, within, or above the clipping interval.
  Why it matters: The negative-advantage case shows how PPO discourages a detrimental action without over-suppressing it.

Key terms: trust region, policy ratio, clip, policy gradient, positive advantage, clipped objective, gradient, negative advantage

6.2.7 Value functions and PPO

Explains how PPO uses a per-token value function as a learned baseline and how simple rollout-based targets train it without generalized advantage estimation.

• PPO value function role: Explain what PPO’s value function predicts and why PPO uses it when updating the policy.
  Why it matters: The baseline explains why PPO needs value predictions alongside its policy update.
• Rollout based critic training: Describe how the section’s simple, non-GAE example obtains value targets and uses the resulting predictions in PPO’s two losses.
  Why it matters: It connects the rollout rewards to both critic training and the signal used for the policy update.

Key terms: value function, learned baseline, return, KL penalties, REINFORCE, Monte Carlo returns, rollouts, value loss, policy loss, advantages

6.2.8 Group relative policy optimization

This section explains how GRPO uses groups of completions to estimate advantages without a learned value function, and examines the resulting tradeoffs and variants.

• Group-based advantage estimation: Explain how GRPO estimates a completion’s advantage by comparing its reward with rewards for other completions of the same prompt, and how that estimate is used in the update.
  Why it matters: Comparing answers to the same prompt is the central idea behind GRPO’s advantage estimate.
• Value-function avoidance: Explain what learned component GRPO avoids relative to PPO and why the section presents that as useful.
  Why it matters: This explains a principal motivation for using GRPO instead of a value-function-based approach.
• Standard-deviation normalization tradeoff: Describe the bias attributed to standard-deviation normalization in GRPO’s advantage and the tradeoff in removing that term.
  Why it matters: The choice of normalization changes which kinds of prompts contribute most strongly to training.
• Outcome and process supervision: Distinguish the GRPO advantage described for outcome supervision from the version needed for process supervision.
  Why it matters: GRPO’s advantage calculation depends on whether feedback scores the outcome or the reasoning process.
• Dr. GRPO and RLOO relation: Explain how removing standard-deviation normalization connects Dr. GRPO’s advantage estimate to RLOO’s estimate.
  Why it matters: It shows that two named algorithms can use closely related advantage estimates.

Key terms: GRPO, group-wise GRPO advantage estimate, completions, rewards, value function, policy language model, reference policy, standard deviation, advantage, low reward variance

6.2.9 Group sequence policy optimization

This section explains why GSPO replaces token-level importance ratios with one length-normalized ratio per response when updating a policy from previously collected rollouts.

• Policy distribution mismatch: Explain why importance sampling is needed when optimizing a current policy using responses generated by an older policy, including how repeated updates or asynchronous training can create the mismatch.
  Why it matters: The distribution mismatch explains why GSPO needs an importance ratio at all.
• Token-level ratio limitations: Describe the learning problems the section attributes to token-level importance ratios and clipping, especially for important tokens and long responses.
  Why it matters: These limitations motivate computing a correction for the whole response instead.
• GSPO sequence-level correction: Explain how GSPO changes GRPO's importance ratio, why it length-normalizes that ratio, and what remains unchanged from GRPO.
  Why it matters: GSPO's defining change is to make the importance-sampling correction operate at the same granularity as the reward.

Key terms: importance sampling, current policy, data-collection policy, distribution mismatch, token-level ratios, clipping, token dropping, GSPO, GRPO, sequence-level importance ratio

6.2.10 Clipped importance sampling policy optimization

This section explains how CISPO clips importance weights while retaining token gradients, and why that tradeoff may help stabilize large-scale RL training.

• CISPO clipping and gradients: Explain how CISPO differs from PPO/GRPO: it clips the importance weight rather than the surrogate objective, uses a stop gradient on that weight, and retains a gradient signal for every token.
  Why it matters: This distinction explains why clipping in CISPO changes the training signal differently from clipping in PPO or GRPO.
• Clipping bias and variance: Explain the tradeoff behind clipping importance weights and why unstable per-token ratios make it relevant to large-scale MoE models.
  Why it matters: It connects CISPO’s clipping choice to a concrete training difficulty rather than treating clipping as an arbitrary change.
• Asymmetric CISPO clipping bounds: Describe how CISPO can use unequal lower and upper clipping bounds and why a higher upper allowance can encourage exploration.
  Why it matters: The bounds can shape which token updates receive more room, not just limit update size uniformly.

Key terms: CISPO, importance weights, stop gradient, surrogate objective, advantage, bias-variance tradeoff, per-token importance sampling ratios, MoE models, asymmetric clipping bounds, exploration

6.2.11 Comparing algorithms

This section compares policy-gradient algorithms by their advantage estimates, importance-sampling strategies, and training requirements, while distinguishing DPO from them.

• Advantage estimation choices: Compare how REINFORCE, RLOO, PPO, and GRPO obtain or adjust the advantage used in their policy-gradient updates.
  Why it matters: The choice of advantage estimate is a principal difference between these algorithms.
• Policy ratio controls: Distinguish which algorithms use an importance-sampling ratio and how PPO, GRPO, CISPO, and GSPO control it.
  Why it matters: Ratio handling explains how the methods differ when updating a policy more than once from a batch of rollouts.
• On-policy and DPO distinction: Explain the on-policy status of the policy-gradient algorithms and how DPO differs from them.
  Why it matters: This distinction keeps a related preference-optimization method from being mistaken for another policy-gradient variant.

Key terms: advantage, baseline, value function, leave-one-out, group, importance-sampling ratio, clipped objective, clipped importance-sampling weights, stop gradient, completion length

6.3 Implementation

The section explains why language-model RL needs implementation choices beyond the original deep-RL algorithms, focusing on generation-length handling and examples of choices that vary across RLHF systems.

• Generation Length and Reward Scoring: Explain why capped generations can cause misleading reward-model scores and how the section says to handle responses that do not finish.
  Why it matters: A generation-length limit can change the feedback used for training, not just the cost of generating text.
• Variable RLHF Implementation Choices: Describe the implementation choices the section identifies for value-network initialization, reward scaling, and KL handling, without treating them as a single mandatory recipe.
  Why it matters: Two implementations of the same named RL algorithm can behave differently because of these choices.

Key terms: EOS token, eos_token, reward model, hard cap, out of distribution, value network initialization, reward normalization, reward whitening, advantage whitening, KL estimators

6.3.1 Policy-gradient basics

This subsection explains how a simple policy-gradient implementation uses a probability ratio and the sign of an advantage to change a model’s token probabilities.

• Per-token probability ratio: Describe the ratio used in the policy-gradient implementation as the new policy’s per-token probability relative to the reference model’s probability, often computed from a log-probability difference.
  Why it matters: This comparison gives meaning to changes in the logratio discussed in the update cases.
• Advantage signs and updates: Explain how positive, negative, and zero advantages affect the policy-gradient update: reinforce an action that was better than expected, make a worse-than-expected completion less likely, and make no update when advantage is zero.
  Why it matters: These three cases show the direction in which the simple policy-gradient loss pushes the model.

Key terms: ratio, per-token probability ratio, new policy model, reference model, log-probability difference, advantage, loss, logratio, policy model

6.3.2 Loss aggregation tradeoffs

This section compares ways to aggregate token losses, explains their length-dependent gradient effects, and connects those choices to sequence-level and token-level views of RLHF.

• Three loss aggregation strategies: Distinguish per-sequence, per-token, and fixed-length normalization by what each gives equal weight to and how sequence length affects the loss.
  Why it matters: The normalization choice changes how much training influence short and long completions have.
• Length-dependent gradient tradeoffs: Explain why equal weighting of completions can produce unequal per-token gradients and why the preferred aggregation can depend on the training setup.
  Why it matters: An apparently fair average over completions can bias learning toward or away from responses of particular lengths.
• Sequence and token RL framings: Distinguish the bandit and MDP framings by their actions, rewards or advantages, and treatment of tokens within a completion.
  Why it matters: The framing clarifies whether an implementation assigns one advantage to a response or distinct advantages to its tokens.
• Undiscounted completion rewards: Justify why RLHF implementations typically set the discount factor to 1.0, even when using a token-level MDP view.
  Why it matters: It explains why treating generation as token-level decisions does not automatically call for standard RL discounting.

Key terms: per-sequence normalization, per-token normalization, fixed-length normalization, L_max, completion_mask, per-token gradients, sequence length, gradient accumulation, numerical stability, variance in loss

6.3.3 Asynchronous RL systems

This section explains why language-model RL systems separate generation from training, how that relaxes exact on-policy execution, and why long, uneven reasoning rollouts complicate the design.

• On-policy execution tradeoff: Explain what exact on-policy execution requires and why practical language-model RL systems do not maintain it exactly.
  Why it matters: It explains why a working training system may depart slightly from the assumptions of the policy-gradient derivation.
• Actor–learner training architecture: Describe how asynchronous systems divide generation and training between actors and learners, and how weights and rollout data connect the two. Distinguish the nearly on-policy setup described here from the proposed fully off-policy approach.
  Why it matters: Separating actors from learners keeps generation and updates running without requiring them to finish in lockstep.
• Uneven rollout length bottlenecks: Explain why long, variable-length reasoning answers can leave compute idle in a synchronous system, and describe the sequence-level packing alternative at the level given here.
  Why it matters: It shows why keeping the generation side busy becomes especially important for reasoning models.

Key terms: on-policy execution, policy gradients, rollouts, actors, learners, nearly on-policy, fully off-policy, weight syncs, reasoning models, sequence-level packing

6.3.4 Truncated importance sampling

This section explains how truncated importance sampling stabilizes language-model RL by capping correction weights, especially when asynchronous sampler and learner systems produce different token probabilities.

• One-sided importance weight cap: Explain how truncated importance sampling reweights samples and why it caps large weights.
  Why it matters: The upper cap explains both the stabilizing benefit of TIS and its difference from clipping a policy ratio around 1.
• Policy drift and backend mismatch: Distinguish the policy-drift correction from the learner–sampler correction and explain when each is needed.
  Why it matters: Separating the two sources of mismatch prevents a reader from treating every importance ratio as a correction for the same problem.
• Single-step and multistep corrections: Trace which correction is active for single-step REINFORCE versus multistep PPO or GRPO, including the effect of recomputing old log-probabilities on the learner.
  Why it matters: Where old log-probabilities come from determines whether backend mismatch receives its own correction or is mixed into the policy ratio.
• Token-level correction bookkeeping: Describe how TIS is applied in practice and distinguish its cap from the numerical clamp.
  Why it matters: The implementation must preserve the right token probabilities to apply the intended correction, rather than mistaking a numerical safeguard for TIS.

Key terms: truncated importance sampling, importance sampling, one-sided upper cap, bounded variance, policy drift, learner–sampler ratio, backend mismatch, effective token distributions, REINFORCE, PPO

6.3.5 Example: PPO

This section walks through a PPO loss computation, explains why clipping stabilizes policy updates, and shows how a single gradient step per sample simplifies PPO and GRPO.

• Old policy and reference model: Distinguish the policy that generated the current batch from the reference model used for the KL penalty.
  Why it matters: These two comparisons serve different purposes in the PPO computation.
• PPO loss computation stages: Trace the main stages of the illustrated PPO computation, including returns, advantages, policy loss, value loss, and masking.
  Why it matters: The clipped policy objective sits within a larger computation that also handles valid tokens and value predictions.
• Pessimistic clipped policy loss: Explain how PPO chooses between the unclipped and clipped policy-gradient losses and why that choice limits policy updates.
  Why it matters: Clipping is the section's central explanation of how PPO stabilizes policy learning.
• Single-step PPO simplification: Explain why PPO or GRPO can omit the second policy-gradient and clipping logic when taking one gradient step per sample.
  Why it matters: The number of gradient steps per sample determines whether this PPO or GRPO implementation needs its clipping machinery.

Key terms: reference RL-policy, old logprobs, reference model, KL distance constraint/penalty, KL penalty, returns, advantages, whitens, value function loss, completion mask

6.3.6 Example: GRPO

This section shows how the GRPO implementation differs from PPO in its advantage computation, value model, and KL penalty placement.

• Group-relative advantage computation: Explain that GRPO computes advantages by normalizing rewards using the mean and standard deviation of completions generated for the same prompt, rather than using a learned value function as a baseline.
  Why it matters: This is the central simplification that distinguishes the example’s GRPO update from PPO.
• KL penalty placement: Distinguish the GRPO example’s addition of a KL penalty directly to the loss from the standard RLHF approach of subtracting it from the reward, while noting that other implementations are possible.
  Why it matters: Where the penalty enters the update is a concrete implementation difference from the PPO example.
• Tokenwise KL tensor shape: Describe why tokenwise KL distances have shape [B, L, N] when a batch contains multiple prompts and multiple completions per prompt.
  Why it matters: The shape connects the tokenwise penalty to the grouped completions used in this implementation.

Key terms: GRPO, group-wise reward statistics, advantages, value model, vf_loss, KL penalty, loss update, reward function, KL distance, [B, L, N]

6.3.7 RLOO vs. GRPO

This section explains how RLOO computes group-relative advantages and why that makes it conceptually similar to GRPO.

• RLOO leave-one-out advantage: Explain how RLOO calculates an advantage for each completion and why this resembles GRPO’s group-relative updates.
  Why it matters: The leave-one-out comparison shows what RLOO and GRPO have in common without treating their other implementation details as identical.

Key terms: RLOO, GRPO, advantage, leave-one-out baseline reward, rloo_k

6.4 Auxiliary topics

6.4.1 Generalized advantage estimation

This section explains how generalized advantage estimation balances bias and variance and how token-level GAE is computed and used in language-model RLHF.

• GAE bias–variance balance: Explain why GAE averages multistep advantage estimates and how lambda changes the balance between bias and variance.
  Why it matters: This explains why GAE uses a mixture of estimation horizons rather than committing to one.
• Backward TD-error accumulation: Describe how the token-level GAE loop accumulates TD errors and why terminal tokens stop that accumulation.
  Why it matters: Terminal handling keeps advantages from one packed sequence from incorporating another sequence’s tokens.
• Value targets and policy weights: Distinguish the GAE outputs used to train the value function from the advantages used in the policy gradient, including why those advantages are detached.
  Why it matters: It clarifies the different training roles of the targets and advantages produced by the GAE calculation.

Key terms: GAE, advantage, bias–variance tradeoff, lambda, multistep advantage estimates, TD residual, TD errors, not_done, GAE accumulator, terminal tokens

6.4.2 Double regularization

Explains how PPO-style update constraints and a KL distance penalty can both regularize RLHF, and why the number of gradient steps per batch affects which constraint matters in practice.

• Two sources of regularization: Distinguish an algorithm’s internal step-size constraint from the KL divergence-based penalty that keeps the fine-tuned policy near its starting policy, and explain why this can make differences between PPO and simpler algorithms less consequential in language-model fine-tuning.
  Why it matters: It helps explain why an algorithmic safeguard important in deep RL may matter less in RLHF.
• Clip fraction as a proxy: Explain what PPO’s clip fraction reports and why a sample counted in it does not necessarily have zero gradient.
  Why it matters: It prevents overinterpreting a common PPO monitoring metric as a count of updates with no gradient.
• Gradient steps and clipping: Explain why PPO-native clipping often does not apply when language-model RLHF uses one gradient step per batch, while multiple update steps can make it relevant; identify the cited multi-step runs as exceptions.
  Why it matters: The number of updates per batch helps determine whether PPO’s own regularizer has a practical role in a training run.

Key terms: step-size constraints, KL divergence-based distance penalty, original policy, PPO, REINFORCE, surrogate objective, clip-fraction, probability ratio, clipping interval, clipped branch

6.5 Further reading

This section points to less-central RLHF optimization alternatives and briefly explains how DAPO and VAPO modify existing approaches.

• Alternative RLHF optimization approaches: Describe how several proposed approaches differ from familiar policy-gradient RLHF methods, while noting that they have not governed best practices.
  Why it matters: It helps distinguish promising variations from the methods the chapter treats as central.
• DAPO changes for reasoning: Explain the four DAPO modifications to GRPO and the training issues they address.
  Why it matters: These changes show how an existing RL update is adapted to the practical demands of long reasoning outputs.
• VAPO value-based combination: Describe how VAPO combines DAPO-style optimizations with value-based techniques.
  Why it matters: It illustrates a different direction for extending GRPO-related training: adding value-based methods.

Key terms: P3O, CoPG, ReMax, MDPO, DAPO, GRPO, dynamic sampling, per-token loss, clip hyperparameters, VAPO

6.6 Summary