The k3 estimator approximates reverse KL from sampled token log probabilities. For each token, let diff be the reference policy’s log probability minus the new policy’s; compute exp(diff) − diff − 1. Unlike the raw log ratio, this estimate is nonnegative and has lower variance than the standard reverse-KL definition. section 11.4
KLt = exp(log πref(yt) − log πθ(yt)) − (log πref(yt) − log πθ(yt)) − 1
Per-token k3 formula used to calculate the KL penalty. section 11.4
