A reward model assigns a scalar score to a prompt–response pair: higher scores represent responses judged more desirable. It acts as a proxy for human preference when improving a language model. section 16.4.1
A typical preference-modeling workflow proceeds as follows:
- Collect prompts and several candidate responses, including handwritten or model-generated responses. section 16.4.1
- Have human evaluators rank each response from most helpful to least helpful. section 16.4.1
- Train a usually smaller Transformer to read the whole sequence and output one floating-point rating. section 16.4.1
- Use predicted rewards to guide reinforcement learning, treating generated responses as actions and scores as feedback. section 16.4.1
Listing 16.15 Pseudocode for the simplest possible RLHF algorithm
for (prompts in dataset) {
responses <- model$generate(prompts) #1
rewards <- reward_model |> predict(responses) #2
good_responses <- responses[rewards > cutoff]
model |> fit(good_responses) #3
}
The pseudocode makes the core loop concrete: generate responses, score them, keep responses above a cutoff, and fit the model on them. section 16.4.1
