chapter ten

10 Fine-tuning large language models with PPO

 

This chapter covers

  • Why large language models need reinforcement learning
  • The three-stage Reinforcement Learning from Human Feedback (RLHF) pipeline
  • Proximal Policy Optimization algorithm theory
  • Building a complete, from-scratch RLHF pipeline for medical patient support chatbot
We are what we repeatedly do. Excellence, then, is not an act, but a habit.

Will Durant, American Historian and Philosopher

Imagine a large hospital system that just deployed an AI chatbot on its patient portal. Millions of patients can now type in their symptoms, ask about medications, or reach out when they're feeling overwhelmed. The chatbot is powered by a large language model — a neural network trained on billions of words from the internet. It can generate fluent, coherent text on virtually any topic. But fluency is not the same as safety.

On its first week, a patient types: "I've been having chest pain for several months. What could this be?" The base model responds with a tidy diagnostic list — coronary artery disease, asthma, acid reflux — in the flat, factual style of an encyclopedia entry. It never tells the patient to seek medical attention immediately. It never acknowledges how frightening persistent chest pain must be. It treats a potentially life-threatening concern like a search engine result.

10.1 A New Frontier for Reinforcement Learning: Language

10.2 Overview of Large Language Models Architecture

10.3 Mapping Language Model to a Markov Decision Process

10.4 The RLHF Pipeline: Three Stages to Alignment

10.5 Supervised Fine-Tuning (SFT)

10.6 Learning a Reward Model from Human Preferences

10.7 Proximal Policy Optimization: From REINFORCE to PPO

10.8 Build a Medical Patient Support Chatbot with RLHF

10.8.1 Pipeline configuration

10.8.2 Safety signals

10.8.3 Stage 1: Supervised fine-tuning

10.8.4 Stage 2: reward model training

10.8.5 Stage 3: PPO fine-tuning

10.8.6 Evaluation

10.9 Summary