Reinforcement learning improves a model using feedback on what it generates. For a language model, the model produces a response, receives a signal about its quality, and uses that signal to guide further training. This is useful when writing every ideal response by hand would be too slow or expensive. section 16.4.1 section 16.4.4
Listing 16.15 Pseudocode for the simplest possible RLHF algorithm
for (prompts in dataset) {
responses <- model$generate(prompts) #1
rewards <- reward_model |> predict(responses) #2
good_responses <- responses[rewards > cutoff]
model |> fit(good_responses) #3
}
This simplified RLHF loop generates responses, scores them with a reward model, and trains on responses above a cutoff. Practical methods use more complex updates that can also learn from lower-scoring responses. section 16.4.1 section 16.4.4
With human feedback, the quality signal can be learned from preferences:
- Start with supervised fine-tuning on handwritten prompts and responses to establish a baseline. section 16.4.1
- Ask people to rank several responses to the same prompt, then train a reward model to assign a score to a prompt–response pair. section 16.4.1
- Use reward scores to guide further training toward preferred responses. section 16.4.1
Feedback without human rankings
A math problem can provide its own check. Generate several answers, mark each final answer with a predictable label, and check it against the known result. Correct responses—including their intermediate work—can become training examples. The same approach can test generated code by running it. Here, answer-checking acts as the environment and generated responses are the actions. section 16.4.4
Verifiable problems also offer a way to continue training on a model’s own output when high-quality human-written pretraining data is scarce. Neither human preferences nor automatic checks are a complete solution to the difficulty of training language models efficiently. section 16.5
