liveBook · Manning

Log-ratio

As explained in Reinforcement Learning from Human Feedback

A log-ratio is the logarithm of the ratio between two positive quantities, such as model and reference probabilities.

A log-ratio measures how much one positive quantity exceeds another on a logarithmic scale. In preference optimization, the key quantity compares a policy’s probability with a frozen reference model’s probability for the same completion: log(πθ(y|x) / πref(y|x)). Summed across completion tokens, it captures the policy’s relative shift while retaining a reference-based constraint. section 8.1.1 section 15.3.1

Direct Preference Optimization (DPO) subtracts the rejected completion’s log-ratio from the chosen completion’s log-ratio, then applies a sigmoid loss scaled by β. Increasing this difference makes the policy prefer the chosen response relative to the reference, while β controls the strength of the preference pull and KL-distance trade-off. section 8.1.1

A practical DPO implementation computes the quantity in a short pipeline:

  • Run policy and frozen-reference forward passes on chosen and rejected completions to obtain sequence log-probabilities. section 8.3
  • Form policy and reference log-ratios by subtracting rejected log-probabilities from chosen log-probabilities. section 8.3
  • Subtract the reference log-ratio from the policy log-ratio, compute the negative log-sigmoid of β times the result, and optionally detach implicit rewards for logging. section 8.3
  • Cache reference log-probabilities when memory is constrained; reuse can reduce peak memory usage by 50%. section 8.3

Chosen and rejected completions

For a prompt x with chosen completion yc and rejected completion yr, DPO compares log(πθ(yc|x)/πref(yc|x)) against log(πθ(yr|x)/πref(yr|x)). A larger chosen-minus-rejected difference lowers the loss; a rejected response receiving the stronger relative shift produces a larger corrective update. section 8.1.1

pi_logratios = policy_chosen_logps - policy_rejected_logps   1
ref_logratios = reference_chosen_logps - reference_rejected_logps2

logits = pi_logratios - ref_logratios3

losses = -F.logsigmoid(beta * logits)4

chosen_rewards = (
    beta * (policy_chosen_logps - reference_chosen_logps).detach()
)5

rejected_rewards = (
    beta * (policy_rejected_logps - reference_rejected_logps).detach()
)

Inspect the compact computation showing policy ratios, reference ratios, their difference, the negative log-sigmoid loss, and detached completion rewards. section 8.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Derives the ratio as an implicit reward, expands it over token log-probabilities, and reaches forward-pass implementation.
Examples
Uses chosen and rejected completions, policy and reference forward passes, DPO equations, and a cacheable implementation listing.
What it assumes
Assumes conditional sequence probabilities and preference pairs, while defining the reference-normalized reward from those quantities.
What you can do after
Apply it via DPO loss computation to compare preference shifts, interpret implicit rewards, and reduce memory with cached reference probabilities.
How it connects
Connects log-ratios to DPO, reference models, KL divergence, reward models, and negative log-likelihood regularization.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.