A patient-support chatbot may give a fluent answer to a distressing question but miss the urgency or fail to show empathy. Reinforcement learning from human feedback (RLHF) guides a language model toward responses people prefer, using a learned reward model to score its behavior. section 10.0 section 10.1
Comparing responses
For a mental health query, “Consider talking to a professional” is plausible but less helpful than a response that acknowledges distress, addresses immediate danger, and suggests appropriate support. A preference comparison can teach a reward model to score the latter more highly. section 10.1 section 10.7
A three-stage RLHF pipeline turns demonstrations and preferences into policy updates:
- Supervised fine-tuning (SFT): Train the language model to imitate curated examples of desirable responses. section 10.9
- Reward modeling: Train a scoring model on pairs of preferred and rejected responses so it learns to rank them. section 10.7
- Reinforcement learning: Generate responses with a trainable policy, score them with the reward model, and update the policy to make higher-scoring responses more likely. section 10.7
Figure 10.3 Reinforcement Learning with Human Feedback pipeline.

The pipeline diagram connects demonstrations, preference modeling, and reinforcement learning. section 10.4
Language generation can be treated as a reinforcement learning episode: the state is the prompt and tokens generated so far, an action selects the next token, and the response ends when an end-of-sequence token is produced. The reward model scores the completed response. Because the main reward arrives at the end, learning can be difficult; reward shaping can provide additional signals. section 10.3
Proximal Policy Optimization (PPO) provides one way to perform the final stage. A policy generates responses, a value head estimates expected reward, and the reward model scores the results. A KL divergence penalty keeps the policy close to an SFT reference model. The reward model is a scalable proxy for human preference, not a fresh human judgment on every response. section 10.7 section 10.8.5 section 10.9
Patient-support evaluation
A small chatbot pipeline uses demonstrations, preference pairs, and PPO with a KL penalty to encourage empathy, appropriate escalation, and avoidance of specific diagnoses. In one higher-compute run, emergency recognition rose from about 79% after SFT to 92% after PPO. Safety, empathy, urgency, and other measures should be assessed separately; the pipeline is not a replacement for clinical review. section 10.8 section 10.8.6 section 10.9
Table 10.2 Safety Metrics Across Training Stages and Compute Configurations
|
Metric
|
Base
|
SFT (Low)
|
PPO (Low)
|
SFT (High)
|
PPO (High)
|
|
Safety
Score
|
0.642–0.657
|
0.732
|
0.758
|
0.733
|
0.780
|
|
Empathy
Rate
|
0.161–0.207
|
0.416–0.430
|
0.472–0.528
|
0.416
|
0.472
|
|
Disclaimer
Rate
|
0.574
|
0.534
|
0.607
|
0.534
|
0.607
|
|
Urgency
Match
|
0.685–0.705
|
0.852–0.882
|
0.869–0.925
|
0.852
|
0.925
|
|
Emergency
Recog.
|
0.500–0.519
|
0.788–0.808
|
0.769–0.923
|
0.788
|
0.923
|
|
Dangerous
Content
|
0.000
|
0.000
|
0.000
|
0.000
|
0.000
|
|
PPO vs Base
Improvement
|
—
|
—
|
+15.5%
|
—
|
+21.6%
|
The results table compares safety measures across training stages and compute configurations. section 10.8.6

















