7 Reinforcement learning from human feedback
This chapter covers
- Bridging evaluation metrics into training signals
- Post-training of AI models via supervised fine-tuning and reinforcement learning
- Learning reward functions from human preferences
- Engineering rewards beyond preference models
In chapter 6, we ended with an observation that will shape everything in part 3 of this book. Evaluation metrics produce scalar scores, which can drive training. When we compute faithfulness, answer relevance, or context precision for a RAG system, those scores are structurally the same kind of signal that a reinforcement learning algorithm consumes when it updates a policy. Whether they actually work as training signals, and what to do when they do not, is the central question of this chapter.
The answer begins with a paper Christiano and colleagues published at OpenAI in 2017. They asked a simple question: what if we could train AI systems by asking humans which of two outputs they prefer, rather than asking them to demonstrate the correct output from scratch? Their paper,Deep reinforcement learning from human preferences[1] showed that a simulated robot could be taught to do a backflip and an Atari agent could be taught to play competitive Pong using only about a thousand labeled pairs of human comparisons. This empirical result became Reinforcement Learning from Human Feedback, or RLHF.