liveBook · Manning

Chapter 10: Fine-tuning large language models with PPO

What you will learn in chapter 10 of Applied Reinforcement Learning

The introduction explains why fluent medical chatbot responses may be unsafe and how reinforcement learning can use human judgments to improve alignment.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why fluent medical chatbot responses may be unsafe and how reinforcement learning can use human judgments to improve alignment.

Key terms: fluency, safety, alignment problem, reinforcement learning, human feedback, next-token training

10.1 A New Frontier for Reinforcement Learning: Language

This section introduces language generation as a reinforcement learning problem and explains why human preferences matter beyond next-token prediction.

Key terms: tokens, actions, environment, reward, human judgment, next-token prediction, statistical patterns, safety, helpfulness, appropriateness

10.2 Overview of Large Language Models Architecture

This section traces how a language model turns text into contextual representations and generates tokens, connecting that process to a stochastic RL policy.

Key terms: pretraining, tokenization, tokens, vocabulary, embeddings, dense vector, position, transformer, attention, feed-forward neural network

10.3 Mapping Language Model to a Markov Decision Process

This section maps language generation to a Markov Decision Process and explains the challenges of rewarding complete responses.

Key terms: state, action, token, vocabulary, transition, policy, stochastic policy, categorical distribution, sampling, reward function

10.4 The RLHF Pipeline: Three Stages to Alignment

This section introduces RLHF as a sequential, three-stage process for aligning language models.

Key terms: RLHF, supervised learning, reward modeling, reinforcement learning

10.5 Supervised Fine-Tuning (SFT)

This section explains how supervised fine-tuning teaches a language model to imitate domain-specific responses, how response-only token loss works, and what imitation alone cannot teach.

Key terms: pre-trained large language model, instruction-tuned, domain-specific goals, supervised fine-tuning, imitation learning, behavioral cloning, query-response pairs, system prompt, user query, ideal response

10.6 Learning a Reward Model from Human Preferences

This section explains how a reward model learns to score medical assistant responses from human preference comparisons and why its behavior depends on the preference data.

Key terms: SFT, reward model, human preferences, reward function, chosen response, rejected response, human preference labels, safety-first, transformer backbone, reward head

10.7 Proximal Policy Optimization: From REINFORCE to PPO

This section explains how PPO uses learned preference rewards to fine-tune a language model while controlling policy updates and drift from the supervised fine-tuned model.

Key terms: policy model, reward model, SFT, token, state, episode, probability ratio, old policy, advantage, clipped objective

10.8 Build a Medical Patient Support Chatbot with RLHF

The section introduces a safety-sensitive medical chatbot case study and explains the safety requirements and practical dataset choices guiding its RLHF pipeline.

Key terms: safety constraints, professional consultation, emergencies, empathy, reward model, reward shaping, final evaluation, custom dataset

10.8.1 Pipeline configuration

This section highlights the decreasing learning rates across the RLHF pipeline stages and relates them to each stage’s role.

Key terms: learning rate, SFT, reward model training, PPO, preference-scoring function, SFT policy

10.8.2 Safety signals

This section describes keyword and pattern checks used as lightweight safety signals in a medical chatbot case study, along with their limitations.

Key terms: emergency language, empathy language, professional referral language, dangerous patterns, reward shaping, evaluation, structured signals, trained reward model, clinical review

10.8.3 Stage 1: Supervised fine-tuning

This section explains the medical chatbot’s supervised fine-tuning stage, its datasets, and the training pipeline.

Key terms: supervised fine-tuning, ideal assistant responses, safety style, label masking, prompt, assistant response, preference dataset, chosen response, rejected response, reward model

10.8.4 Stage 2: reward model training

This section explains how the medical chatbot’s reward model and supplementary shaping signals are constructed, trained, and validated.

Key terms: reward shaping, shaping rewards, learned reward, reward model, query, response, last token's hidden state, causal attention, reward head, pairwise ranking loss

10.8.5 Stage 3: PPO fine-tuning

This section explains how the medical chatbot’s PPO stage collects experience, computes advantages, updates its policy, and evaluates safety.

Key terms: policy model, value head, reference model, reward model, PPO trainer, single-turn, return, reward, advantage, value

10.8.6 Evaluation

Evaluates the medical chatbot’s RLHF results and illustrates how PPO supports stable policy improvement beyond language models.

Key terms: SFT, PPO, emergency recognition, urgency match, low-compute run, high-compute run, scaling law, alignment, dangerous content rate, base model

10.9 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 9: The Calculus of Decisions: Policy Gradient Methods Chapter 11: Teach LLMs to reason with Verifiable Feedback using GRPO →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.