How deeply does Reinforcement Learning from Human Feedback cover difficulty filtering?

Defines the signal rationale and gives sampling, verification, solve-rate, and curriculum procedures, but does not specify an optimal recipe.

Difficulty filtering

As explained in Reinforcement Learning from Human Feedback

Difficulty filtering is a training-data selection technique that retains or emphasizes examples according to their estimated difficulty.

Difficulty filtering selects training problems by how often a starting model solves them. In RLVR, problems solved almost always or never provide little policy-gradient signal, so training commonly retains problems solved 20–80% of the time. section 7.3.3 section 7.5

A practical offline filtering pipeline is:

Sample multiple completions for every candidate prompt, such as 16 responses. section 7.3.3
Verify each completion with a correctness signal, such as answer matching or another verifier. section 7.3.3 section 7.5
Estimate difficulty from the fraction of correct completions and retain prompts within the desired range, commonly 20–80%. section 7.3.3

Offline filtering chooses the training pool before large-scale RL. Online filtering or a difficulty curriculum instead selects problems during training, presents suitable questions per batch, or postpones harder problems to later stages for greater stability. section 7.3.3

Why the middle matters

A prompt the starting policy answers correctly on every sampled completion offers no useful contrast between strategies; a prompt it never answers correctly offers no successful behavior to reinforce. A prompt with mixed correct and incorrect completions supplies the variation needed for policy-gradient learning. section 7.3.3

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

completionuses · A completion is content associated with a language-model prompt, either as an expected target or as a generated continuation or response.
reinforcement learning with verifiable rewardsused by · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.