How deeply does Reinforcement Learning from Human Feedback cover length normalization?

Compares three normalization strategies, derives their objectives, and illustrates gradient weighting with two sequences.

Length normalization

As explained in Reinforcement Learning from Human Feedback

Length normalization rescales an objective, score, or gradient according to the number of tokens or steps in an output.

Length normalization controls how per-token losses become a scalar objective in language-model policy-gradient training. Fixed-length normalization divides each sequence’s active-token loss by a shared maximum generation length, L_max, then averages sequences. section 6.3.2

The denominator determines whose gradient influence grows with sequence length:

Per-sequence normalization weights sequences equally, giving tokens in shorter sequences larger gradients. section 6.3.2
Per-token normalization weights active tokens equally, so longer sequences contribute more total gradient. section 6.3.2
Fixed-length normalization equalizes per-token scale across sequences while retaining greater total influence for longer sequences. section 6.3.2

Two completions

For five-token losses [1, 1, 1, 1, 10] and ten-token losses [1, 1, 1, 1, 1, 1, 1, 1, 1, 10], fixed-length normalization with L_max = 10 contributes 1.4 and 1.9 respectively. section 6.3.2

Equation 6.48

\[\label{eq:loss_fixed_length} L = \frac{1}{B} \sum_{i=1}^{B} \frac{1}{L_{\max}} \sum_{t=1}^{|a_i|} \ell_{i,t}.\]

The equation makes the shared denominator and sequence averaging explicit. section 6.3.2

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

length biasaddresses · Length bias is a systematic preference or scoring distortion based on output length or verbosity, independent of output quality or usefulness.
loss aggregationpart of · Loss aggregation is the method of combining per-token or per-sequence losses into a scalar optimization loss.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.