liveBook · Manning

Online filtering

As explained in Reinforcement Learning from Human Feedback

Online difficulty filtering is the selection of training problems during optimization according to their current difficulty or learning signal.

Online filtering selects training problems during optimization according to their current difficulty or learning signal. It targets prompts whose sampled completions provide useful differences in policy-gradient outcomes. section 7.3.3

A practical filtering pipeline can:

  • Sample multiple completions for each prompt, verify correctness, and estimate its current success rate. section 7.3.3
  • Prefer problems with mixed outcomes, such as 20–80% correctness, rather than prompts solved always or never. section 7.3.3
  • Filter questions within each batch or schedule harder problems later through curricula and data schedulers. section 7.3.3

Batch-level filtering

For each training prompt, sample 16 completions and verify them. Retain prompts whose current correctness rate supplies contrasting outcomes, then assemble the optimization batch. section 7.3.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Explains sampling, correctness estimation, batch filtering, and curricula, but does not specify a complete filtering algorithm.
Examples
Uses 16 sampled completions, 20–80% correctness ranges, per-batch filtering, and deferred harder problems.
What it assumes
Assumes reinforcement-learning optimization, policy gradients, sampled completions, and correctness verification.
What you can do after
Apply it by estimating prompt difficulty online and selecting batches or curricula that preserve informative outcome variation.
How it connects
Connects difficulty filtering with policy gradients, reinforcement learning from verifiable rewards, curricula, and data scheduling.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.