liveBook · Manning

Length penalty

As explained in Reinforcement Learning from Human Feedback

A length penalty is an objective term that reduces the optimization preference for outputs with undesired lengths.

A length penalty is a soft objective term that discourages excessively long outputs. In reasoning-language-model reinforcement learning, it reduces the incentive to learn from answers truncated by maximum-length limits. section 6.5

Operationally, the penalty:

  • Targets samples that are too long rather than imposing a hard exclusion. section 6.5
  • Avoids learning from truncated answers, which may provide a poor training signal. section 6.5
  • Fits into existing reinforcement-learning infrastructure as one modification used by DAPO for long reasoning traces. section 6.5

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Explains the penalty’s operational purpose and integration point, without specifying its mathematical form or tuning procedure.
Examples
Uses overly long samples and truncated answers as the concrete failure case motivating the penalty.
What you can do after
Understand how to discourage learning from excessively long or truncated reasoning outputs within an RL pipeline.
How it connects
Connects length penalties to DAPO, GRPO, reinforcement learning, policy optimization, and truncated answers.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.