Length bias

Q: How deeply does Reinforcement Learning from Human Feedback cover length bias?

Identifies length-driven scoring and preference distortions, while deferring detailed mitigation treatment.

As explained in Reinforcement Learning from Human Feedback

Length bias is a systematic preference for outputs of particular lengths that can distort model optimization or evaluation.

Length bias is a systematic preference for outputs of particular lengths. In RLHF and preference-based training, it can distort optimization when a proxy reward favors longer responses rather than better ones. In evaluation, the same distortion appears when an LLM judge assigns higher scores to longer answers regardless of quality. section 1.2 section 16.0

Length bias enters post-training pipelines in several places:

Reward optimization can over-optimize a proxy signal, making output length part of the learned preference rather than a reliable quality indicator. section 1.2
Synthetic preference data can inherit the judging model’s bias, affecting which completions are labeled chosen or rejected. section 8.4
Chat benchmarks using LLM judges can reward verbosity independently of response quality; length-controlled variants emerged in response. section 16.0

A biased evaluation signal

Two answers may differ in length without the longer answer being better. If an LLM judge consistently scores the longer response higher, the benchmark reports a misleading quality difference and can encourage models to produce unnecessarily long outputs. section 16.0

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

length normalizationaddressed by · Length normalization adjusts a score, objective, gradient, or comparison to account for differences in sequence or output length.
Direct Preference Optimizationproduced by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
GSPOaddressed by · GSPO is a policy-optimization algorithm that computes length-normalized importance ratios over complete response sequences rather than individual tokens.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.