Reinforcement learning with human feedback

As explained in LLMs in Production

Reinforcement learning from human feedback is a model-training approach that optimizes behavior using preference or evaluation signals supplied by people.

A game-playing bot can learn the wrong lesson from a reasonable-sounding reward. Reward it for building a useful structure, and it may keep building instead of winning. Reinforcement learning with human feedback (RLHF) addresses this reward-design problem by asking people which attempts they prefer. For a language model, a reviewer compares responses to a prompt and selects the better one. section 5.3.3

The preference signal guides training toward outputs human evaluators like. A reward model and proximal policy optimization (PPO) provide the training setup shown here, rather than relying only on a conventional loss function. Preference is not the same as correctness: RLHF can make responses more pleasing while increasing factually incorrect results. section 5.3.3 section 5.5

When reviewers see user prompts

In AI Dungeon, access to a GPT-3 model using RLHF meant OpenAI contractors could see prompts. Concerns about the game’s content led to demands for stronger filtering; the resulting safeguards frustrated players. Human review can therefore become a privacy and product-policy issue, not just a training expense. section 1.2.2

RLHF is powerful, but it is computationally expensive and may deliver only incremental gains. It can also require paid domain experts to evaluate interactions, raising cost and privacy concerns. Before choosing it, consider whether better-quality data and supervised learning could achieve a comparable result. section 5.3.3

Figure 5.6 RLHF substitutes a loss function for a reward model and proximal policy optimization (PPO), allowing the model a much higher ceiling for learning trends within the data, including what is preferred as an output instead of what completes the task.

The diagram connects human-preferred outputs to the reward model and PPO used in RLHF training. section 5.3.3

How LLMs in Production covers it

How deep it goes

Explains preference selection through examples and identifies reward models and PPO, without working through their implementation.

Examples

Uses a StarCraft bot exploiting an intermediate reward, response comparisons, and AI Dungeon’s content-filtering dispute.

What it assumes

Introduces reward-design difficulty through a game example before applying human preference comparisons to language models.

What you can do after

Evaluate/understand it as a preference-training option against supervised learning, accounting for compute, reviewers, privacy, and factual errors.

How it connects

Connects reinforcement learning, reward design, reward models, PPO, supervised learning, and factual accuracy.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
• 

  liveBookmanning.com
  How Large Language Models WorkReinforcement learning from human feedback is a technique that optimizes a model's behavior using human evaluations as a source of reward or preference information.
• 

  liveBookmanning.com
  Generative AI in ActionReinforcement learning from human feedback is a model-training technique that uses human preferences as feedback to optimize generated behavior.
• 

  liveBookmanning.com
  The AI Pocket BookA training method that uses human judgments as feedback to guide reinforcement learning and improve model outputs.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A training approach that uses human preference labels to optimize a language model toward responses humans prefer.
• 

  liveBookmanning.com
  Grokking AI ApplicationsReinforcement learning from human feedback is a training technique that optimizes model behavior using human evaluations of its responses.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionReinforcement learning with human feedback is a model-training technique that uses human evaluations to optimize a system's behavior.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionReinforcement learning from human feedback (RLHF) is a technique that uses human evaluations or preferences to estimate rewards and optimize model behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionReinforcement learning with human feedback is a training approach that optimizes a model using human preference judgments converted into reward signals.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training setup that uses human preference feedback to build a reward signal for improving a model’s outputs.
• 

  liveBookmanning.com
  Transformers in ActionReinforcement learning from human feedback is a model-optimization technique that uses human preference feedback to define or improve a reward signal.
• 

  liveBookmanning.com
  First-Party Data ActivationReinforcement learning from human feedback is a model-training technique that uses human evaluations to shape a system’s behavior.
• 

  liveBookmanning.com
  Software Testing with Generative AIA model-alignment technique that uses human feedback to guide or refine a model's behavior.

Compare how 14 books explain reinforcement learning with human feedback →

Related concepts

• reward modeluses · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
• Proximal Policy Optimizationuses · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• reinforcement learninguses · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• Direct Preference Optimizationcontrasts with · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
• RLAIFalternative to · Reinforcement learning from AI feedback is a training or alignment technique that uses AI-generated evaluations or preferences as feedback to optimize a model or agent.
• ChatGPTused by · ChatGPT is OpenAI’s conversational AI service, which lets users interact with generative AI models and related tools through natural-language prompts.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.