liveBook · Manning

KL regularization

As explained in Build a DeepSeek Model (From Scratch)

KL control is the regulation of policy updates by penalizing divergence between the learned policy and a reference policy.

KL regularization keeps a trainable policy near a frozen reference policy while reward optimization changes its behavior. The reference policy is typically a copy of the model before reinforcement learning and supplies a stable anchor against reward exploitation, language-quality degradation, and behavioral drift. In PPO-style language-model training, the system compares token-level log probabilities for generated response tokens; larger divergence adds a penalty. section 7.3.3 section 7.3.1

Implement KL control as part of the rollout and loss pipeline:

  • Freeze the reference model and retain it throughout the reinforcement-learning stage. section 7.12
  • Run the trainable and reference policies on identical generated sequences, gathering log probabilities only for response tokens. section 7.12
  • Mask prompt and padding tokens so they contribute neither to the KL term nor to the policy loss. section 7.14
  • Combine the KL penalty with probability-ratio clipping and group-relative advantages in the GRPO objective. section 7.12
  • Monitor mean KL alongside reward, response length, invalid-format rate, and loss; rising reward with exploding KL signals instability. section 7.12

Choosing KL pressure

Set KL pressure too low and the policy may drift toward exploitable reward quirks; set it too high and learning can stall. Practical systems adjust pressure using observed divergence rather than treating the coefficient as permanently fixed. section 7.14

Step
Question to ask before scaling
Task
Can the outcome be verified reliably?
Format
Can the final answer be parsed without ambiguity?
Rollout
Are old log probabilities and response masks stored correctly?
Reward
Are parse failures and invalid outputs tracked separately?
Advantage
Does each prompt group have useful reward variance?
KL
Is the policy staying close enough to the frozen reference?
Evaluation
Does held-out performance improve, not just training reward?

Use the implementation checklist to verify task correctness, response formatting, rollout storage, reward diagnostics, group variance, KL stability, and held-out evaluation before scaling. section 7.12

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the frozen anchor, token-level log-probability comparison, masking, monitoring, and pressure trade-offs, but not a full optimizer implementation.
Examples
Uses frozen pre-training copies, generated response tokens, attention masks, log probabilities, and reward-KL diagnostic metrics.
What it assumes
Assumes familiarity with policy optimization, log probabilities, reward signals, PPO, and GRPO while distinguishing reference and value models.
What you can do after
Operate/configure KL control by freezing a reference policy, masking tokens, computing divergence, tuning pressure, and monitoring stability.
How it connects
Connects KL regularization to PPO, GRPO, reward models, verifiers, policy drift, and value models.

Also explained in

Compare how 5 books explain KL regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.