liveBook · Manning

Progressive length scheduling

As explained in Reinforcement Learning from Human Feedback

Progressive length scheduling is a training procedure that changes the targeted or permitted output length over successive optimization stages.

Progressive length scheduling changes the targeted or permitted output length across successive optimization stages. In reasoning training, it helps limit overthinking while preserving the longer reasoning chains associated with improved performance. section 7.5

Use the technique within reasoning-training pipelines alongside:

  • Inference-time scaling, where additional generation computation can improve responses. section 7.5
  • Difficulty filtering, exploration-friendly KL choices, and format or language consistency rewards. section 7.5

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Identifies the scheduling purpose but does not specify its stages, length thresholds, or optimization implementation.
What it assumes
Assumes familiarity with reasoning training, policy optimization, and reinforcement-learning terminology.
What you can do after
Apply progressive output-length control during reasoning training to combat overthinking.
How it connects
Links reasoning training with inference-time scaling, difficulty filtering, KL penalties, RLHF, and reinforcement learning from verifiable rewards.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.