liveBook · Manning

Implicit reward

As explained in Reinforcement Learning from Human Feedback

An implicit reward is a reward value inferred from a preference-optimization objective rather than predicted by a separately trained reward model.

An implicit reward model assigns reward through a preference-optimization objective instead of predicting reward with a separately trained reward model. In DPO, the reward for prompt–completion pair (x, y) is the policy’s log-probability relative to a frozen reference model: r(x, y) = β log[π(y|x) / πref(y|x)]. This expresses preference evidence through the optimal policy rather than an external reward-model training stage. section 8.1.1

Operationally, the implicit reward connects model behavior to the DPO training pipeline:

  • Computes policy and reference log-probabilities for chosen and rejected completions. section 8.3
  • Forms each completion’s policy-to-reference log-ratio, then compares the chosen and rejected ratios. section 8.1.1
  • Weights updates more heavily when the rejected completion receives the higher implicit reward. section 8.1.1
  • Uses β to balance preference optimization against divergence from the reference model. section 8.1.1

Chosen versus rejected completion

For a prompt x, DPO increases the gap between the chosen completion’s relative log-probability and the rejected completion’s relative log-probability. The loss is -log σ(β[(log πθ(yc|x) - log πref(yc|x)) - (log πθ(yr|x) - log πref(yr|x))]). The resulting detached chosen_rewards and rejected_rewards can be logged as implicit rewards without training a separate reward model. section 8.1.1 section 8.3

Equation 8.2
\[\label{eq:dpo_core} \mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_{\theta}(y_c \mid x)}{\pi_{\text{ref}}(y_c \mid x)} - \beta \log \frac{\pi_{\theta}(y_r \mid x)}{\pi_{\text{ref}}(y_r \mid x)} \right) \right].\]

Displays the core DPO loss, which compares policy-to-reference log-ratios for chosen and rejected completions. section 8.1.1

Equation 8.3
\[\label{eq:dpo_reward} r(x, y) = \beta \log \frac{\pi_r(y \mid x)}{\pi_{\text{ref}}(y \mid x)},\]

Displays the implicit reward definition as a scaled policy-to-reference log-ratio. section 8.1.1

pi_logratios = policy_chosen_logps - policy_rejected_logps   1
ref_logratios = reference_chosen_logps - reference_rejected_logps2

logits = pi_logratios - ref_logratios3

losses = -F.logsigmoid(beta * logits)4

chosen_rewards = (
    beta * (policy_chosen_logps - reference_chosen_logps).detach()
)5

rejected_rewards = (
    beta * (policy_rejected_logps - reference_rejected_logps).detach()
)

Shows implementation pseudocode for log-ratio differences, the negative log-sigmoid loss, and detached completion rewards. section 8.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Derives the reward, loss, and gradient, then maps them to forward-pass log-probabilities and detached logging; it defers broader reward-model training details.
Examples
Uses chosen and rejected completions, policy and reference log-probabilities, DPO equations, and implementation-style reward variables.
What it assumes
Assumes language-model log-probabilities, chosen–rejected preference pairs, a reference model, and familiarity with KL regularization.
What you can do after
Apply it via DPO to compute preference losses and monitor completion-level rewards without maintaining an external reward model.
How it connects
Connects DPO, reference models, KL divergence, preference optimization, Bradley–Terry models, and RLHF.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.