Reinforcement learning lets a model improve through interaction: it generates responses, receives feedback or rewards, and favors behavior associated with higher scores. For language models, the practical motivation is to optimize preferences that are difficult to express directly as a loss function. section 16.4.1 section 16.5
A simplified RLHF workflow turns human preference into a training signal:
- Start with supervised fine-tuning on handwritten prompts and responses to establish a baseline. section 16.4.1
- Collect several responses for prompts and have evaluators rank them from most helpful to least helpful. section 16.4.1
- Train a reward model that reads a prompt-response pair and returns a floating-point preference score. section 16.4.1
- Use the reward model to score generated responses and tune the language model toward higher-reward behavior. section 16.4.1
A minimal reward-filtering loop
For each prompt batch, generate responses, predict their rewards, retain responses above a cutoff, and fit the model on the retained set. This pseudocode captures the feedback loop without developing the deeper reinforcement-learning machinery. section 16.4.1
The same idea can extend beyond preference rankings: a difficult, self-contained environment can continually generate new problems for an LLM, allowing training from model interaction rather than an ever-larger supply of public text. section 16.5
