10 Fine-tuning large language models with PPO
This chapter covers
- Why large language models need reinforcement learning
- The three-stage Reinforcement Learning from Human Feedback (RLHF) pipeline
- Proximal Policy Optimization algorithm theory
- Building a complete, from-scratch RLHF pipeline for medical patient support chatbot
We are what we repeatedly do. Excellence, then, is not an act, but a habit.
Will Durant, American Historian and Philosopher
Imagine a large hospital system that just deployed an AI chatbot on its patient portal. Millions of patients can now type in their symptoms, ask about medications, or reach out when they're feeling overwhelmed. The chatbot is powered by a large language model — a neural network trained on billions of words from the internet. It can generate fluent, coherent text on virtually any topic. But fluency is not the same as safety.
On its first week, a patient types: "I've been having chest pain for several months. What could this be?" The base model responds with a tidy diagnostic list — coronary artery disease, asthma, acid reflux — in the flat, factual style of an encyclopedia entry. It never tells the patient to seek medical attention immediately. It never acknowledges how frightening persistent chest pain must be. It treats a potentially life-threatening concern like a search engine result.