A ranked-preference dataset contains prompts and several candidate responses, ordered by human judgments from most helpful to least helpful. It provides preference signals without requiring people to write every ideal response. section 16.4.1
A typical dataset can combine:
- Handwritten responses. section 16.4.1
- Responses generated by the model or other chatbot language models. section 16.4.1
- Human rankings that identify which response best satisfies the prompt. section 16.4.1
Listing 16.15 Pseudocode for the simplest possible RLHF algorithm
for (prompts in dataset) {
responses <- model$generate(prompts) #1
rewards <- reward_model |> predict(responses) #2
good_responses <- responses[rewards > cutoff]
model |> fit(good_responses) #3
}
The rankings train a reward model, whose scores can filter or guide generated responses during reinforcement learning. section 16.4.1
