liveBook · Manning

RLHF

As explained in Applied Reinforcement Learning

Reinforcement learning from human feedback uses human feedback, often through a learned preference model, to guide a model's behavior.

A patient-support chatbot may give a fluent answer to a distressing question but miss the urgency or fail to show empathy. Reinforcement learning from human feedback (RLHF) guides a language model toward responses people prefer, using a learned reward model to score its behavior. section 10.0 section 10.1

Comparing responses

For a mental health query, “Consider talking to a professional” is plausible but less helpful than a response that acknowledges distress, addresses immediate danger, and suggests appropriate support. A preference comparison can teach a reward model to score the latter more highly. section 10.1 section 10.7

A three-stage RLHF pipeline turns demonstrations and preferences into policy updates:

  • Supervised fine-tuning (SFT): Train the language model to imitate curated examples of desirable responses. section 10.9
  • Reward modeling: Train a scoring model on pairs of preferred and rejected responses so it learns to rank them. section 10.7
  • Reinforcement learning: Generate responses with a trainable policy, score them with the reward model, and update the policy to make higher-scoring responses more likely. section 10.7
Figure 10.3 Reinforcement Learning with Human Feedback pipeline.

The pipeline diagram connects demonstrations, preference modeling, and reinforcement learning. section 10.4

Language generation can be treated as a reinforcement learning episode: the state is the prompt and tokens generated so far, an action selects the next token, and the response ends when an end-of-sequence token is produced. The reward model scores the completed response. Because the main reward arrives at the end, learning can be difficult; reward shaping can provide additional signals. section 10.3

Proximal Policy Optimization (PPO) provides one way to perform the final stage. A policy generates responses, a value head estimates expected reward, and the reward model scores the results. A KL divergence penalty keeps the policy close to an SFT reference model. The reward model is a scalable proxy for human preference, not a fresh human judgment on every response. section 10.7 section 10.8.5 section 10.9

Patient-support evaluation

A small chatbot pipeline uses demonstrations, preference pairs, and PPO with a KL penalty to encourage empathy, appropriate escalation, and avoidance of specific diagnoses. In one higher-compute run, emergency recognition rose from about 79% after SFT to 92% after PPO. Safety, empathy, urgency, and other measures should be assessed separately; the pipeline is not a replacement for clinical review. section 10.8 section 10.8.6 section 10.9

Table 10.2 Safety Metrics Across Training Stages and Compute Configurations
Metric
Base
SFT (Low)
PPO (Low)
SFT (High)
PPO (High)
Safety Score
0.642–0.657
0.732
0.758
0.733
0.780
Empathy Rate
0.161–0.207
0.416–0.430
0.472–0.528
0.416
0.472
Disclaimer Rate
0.574
0.534
0.607
0.534
0.607
Urgency Match
0.685–0.705
0.852–0.882
0.869–0.925
0.852
0.925
Emergency Recog.
0.500–0.519
0.788–0.808
0.769–0.923
0.788
0.923
Dangerous Content
0.000
0.000
0.000
0.000
0.000
PPO vs Base Improvement
—
—
+15.5%
—
+21.6%

The results table compares safety measures across training stages and compute configurations. section 10.8.6

How Applied Reinforcement Learning covers it

How deep it goes
Works through demonstrations, pairwise reward modeling, PPO updates, and evaluation, while omitting routine trainer utilities.
Examples
Uses preferred mental-health responses and a medical patient-support chatbot with safety and emergency-recognition metrics.
What you can do after
Builds an illustrative RLHF pipeline from scratch for patient support, without replacing clinical review.
How it connects
Connects supervised fine-tuning, reward modeling, Markov decision processes, PPO, and KL divergence.

Also explained in

Compare how 18 books explain RLHF →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.