KL regularization keeps a trainable policy near a frozen reference policy while reward optimization changes its behavior. The reference policy is typically a copy of the model before reinforcement learning and supplies a stable anchor against reward exploitation, language-quality degradation, and behavioral drift. In PPO-style language-model training, the system compares token-level log probabilities for generated response tokens; larger divergence adds a penalty. section 7.3.3 section 7.3.1
Implement KL control as part of the rollout and loss pipeline:
- Freeze the reference model and retain it throughout the reinforcement-learning stage. section 7.12
- Run the trainable and reference policies on identical generated sequences, gathering log probabilities only for response tokens. section 7.12
- Mask prompt and padding tokens so they contribute neither to the KL term nor to the policy loss. section 7.14
- Combine the KL penalty with probability-ratio clipping and group-relative advantages in the GRPO objective. section 7.12
- Monitor mean KL alongside reward, response length, invalid-format rate, and loss; rising reward with exploding KL signals instability. section 7.12
Choosing KL pressure
Set KL pressure too low and the policy may drift toward exploitable reward quirks; set it too high and learning can stall. Practical systems adjust pressure using observed divergence rather than treating the coefficient as permanently fixed. section 7.14
|
Step
|
Question to ask before scaling
|
|
Task
|
Can the outcome be verified reliably?
|
|
Format
|
Can the final answer be parsed without ambiguity?
|
|
Rollout
|
Are old log probabilities and response masks stored
correctly?
|
|
Reward
|
Are parse failures and invalid outputs tracked separately?
|
|
Advantage
|
Does each prompt group have useful reward variance?
|
|
KL
|
Is the policy staying close enough to the frozen
reference?
|
|
Evaluation
|
Does held-out performance improve, not just training
reward?
|
Use the implementation checklist to verify task correctness, response formatting, rollout storage, reward diagnostics, group variance, KL stability, and held-out evaluation before scaling. section 7.12




