liveBook · Manning

Sigmoid

As explained in Reinforcement Learning from Human Feedback

The sigmoid function maps a real-valued input to a number between zero and one using the function 1/(1+e^-x).

The sigmoid function converts any real-valued input into a number between zero and one: σ(z) = 1/(1 + e^-z). In pairwise preference modeling, it turns a score difference into a preference probability. section 5.1

For reward-model scores, σ(rθ(yc|x) − rθ(yr|x)) represents the probability that the chosen completion is preferred to the rejected completion. Larger positive differences approach one; negative differences approach zero. section 5.1

Pairwise reward modeling

Given a prompt x and completions yc and yr, maximize σ(rθ(yc|x) − rθ(yr|x)), or equivalently minimize −log σ(rθ(yc|x) − rθ(yr|x)). section 5.1

Equation 5.2
\[\label{eq:bradterry_unbounded} {P(i > j) = \frac{e^{r_i}}{e^{r_i} + e^{r_j}} = \sigma(r_i-r_j).}\]

The Bradley–Terry equation shows how exponentiated unbounded scores reduce to a sigmoid of their difference. section 5.1

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the function and derives its use in equivalent likelihood and negative-log-likelihood objectives.
Examples
Uses chosen and rejected language-model completions scored under a prompt.
What it assumes
Assumes pairwise preferences and scalar scores while defining sigmoid notation directly.
What you can do after
Apply sigmoid within a Bradley–Terry reward-model objective for pairwise preference training.
How it connects
Ties sigmoid to the Bradley–Terry model, reward models, pairwise preferences, and negative log-likelihood.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

Compare how 16 books explain sigmoid →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.