liveBook · Manning

Policy model

As explained in Applied Reinforcement Learning

A policy model is the trainable model that generates actions according to a policy.

In reinforcement learning from human feedback (RLHF), the policy model is the trainable copy of a supervised fine-tuning model. It generates a response token by token, receives a score from the reward model, and updates its response probabilities through PPO. section 10.7

Medical chatbot

For a chest-pain prompt, the policy can learn to favor a response that acknowledges concern, avoids diagnosis, recommends medical evaluation, and escalates severe symptoms over one that only lists possible causes. section 10.7

Figure 10.6 Reinforcement Learning from Human Feedback with PPO.

The pipeline places the policy model in the RLHF stage, after supervised fine-tuning and reward modeling. section 10.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines the model’s role and token-by-token generation, but does not detail its implementation.
Examples
Contrasts chest-pain responses that merely list causes with responses that advise care and escalate severe symptoms.
What it assumes
Starts from a supervised fine-tuning model and uses a previously trained reward model.
What you can do after
Understand how PPO uses reward-model scores to make preferred generated responses more likely.
How it connects
Connects supervised fine-tuning, reward models, reinforcement learning from human feedback, and proximal policy optimization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.