liveBook · Manning

Schulman k3 estimator

As explained in Applied Reinforcement Learning

A nonnegative Monte Carlo estimator of reverse Kullback–Leibler divergence computed from differences between log probabilities.

The k3 estimator approximates reverse KL from sampled token log probabilities. For each token, let diff be the reference policy’s log probability minus the new policy’s; compute exp(diff) − diff − 1. Unlike the raw log ratio, this estimate is nonnegative and has lower variance than the standard reverse-KL definition. section 11.4

KLt = exp(log πref(yt) − log πθ(yt)) − (log πref(yt) − log πθ(yt)) − 1

Per-token k3 formula used to calculate the KL penalty. section 11.4

How Applied Reinforcement Learning covers it

How deep it goes
Defines the per-token formula and its nonnegativity and variance properties.
Examples
Uses a per-token log-probability difference and exponential expression as the calculation.
What it assumes
Uses reference-policy and new-policy token log probabilities to form the estimate.
What you can do after
Apply it in GRPO or PPO to estimate reverse KL from sampled policy log probabilities.
How it connects
Connects reverse KL estimation to Monte Carlo methods, REINFORCE, GRPO, and PPO.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.