liveBook · Manning

Kullback–Leibler divergence

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Kullback-Leibler divergence is a measure of how one probability distribution differs from a reference probability distribution.

Kullback–Leibler (KL) divergence measures how much one probability distribution differs from another. In reinforcement learning from human feedback, it measures how far an RL-tuned policy has drifted from its supervised fine-tuning (SFT) reference. section 7.5.4

A KL penalty turns this comparison into a practical control loop:

  • Start with the raw reward-model score for a prompt–response pair. section 7.5.4
  • Subtract a small positive coefficient times the KL divergence between the current policy and the reference policy. section 7.5.4
  • Optimize the resulting reward, trading reward-model improvement against policy drift. section 7.5.4

Why the penalty matters

A reward model is calibrated on SFT-model outputs. If PPO moves far from that distribution, the reward model may be unreliable and reward hacking can produce repetitive sequences or gibberish with high scores. KL discourages leaving the region where those scores are meaningful. section 7.5.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines distributional difference and explains its reward penalty, but does not derive the divergence mathematically.
Examples
Uses repetitive token sequences, gibberish, and teacher–student token-probability matching as concrete contexts.
What it assumes
Assumes probability distributions and policy optimization; explains the alignment motivation for the penalty.
What you can do after
Apply it via PPO to balance reward-model gains against changes from a reference policy.
How it connects
Connects KL divergence to PPO, reward models, supervised fine-tuning, and logit-based distillation.

Also explained in

Compare how 19 books explain Kullback–Leibler divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.