Reinforcement learning with human feedback

As explained in How Large Language Models Work

Reinforcement learning from human feedback is a technique that optimizes a model's behavior using human evaluations as a source of reward or preference information.

Reinforcement learning from human feedback (RLHF) changes an LLM’s behavior by using human judgments as rewards. It is more complex than supervised fine-tuning but can target abstract goals such as being a helpful chatbot, rather than only learning new information or a domain style. section 5.2 section 5.6

Figure 5.4 RL is about iterative interactions, where the reward for your actions may not materialize for a long time and requires multiple steps to achieve. For a chatbot like ChatGPT, the environment is the conversation with a user, and the actions are the infinite possible texts that ChatGPT might complete. The reward becomes, in some sense, the user’s satisfaction with the chatbot at the end of the conversation.

Use the interaction diagram to trace the LLM as agent, generated text as action, conversation as environment, and user satisfaction as a delayed reward. section 5.2.2

A high-level RLHF process combines several ingredients:

• Define the interaction: the LLM is the agent, its possible text outputs are actions, and the conversation with a user is the environment. section 5.2.2
• Collect human judgments, such as positive scores for helpful, honest conversations and negative scores for harmful or poor responses. section 5.2.2 section 5.3.1
• Use generated-token probabilities to give the reinforcement-learning algorithm more information than a simple good-or-bad label. section 5.3.1
• Train a reward model so human feedback can be reused to score many prompts instead of requiring a person to judge every new output. section 5.3.3
• Balance quality against similarity to the original model; otherwise, maximizing the quality score alone can produce gibberish or nonsensical outputs. section 5.3.3

Scoring a joke

Given the prompt “Tell me a joke,” a joke-like response might receive +1, while “Dogs are evil” might receive -1 because it does not attempt the task. Such coarse scores illustrate both the usefulness and subjectivity of human rewards; token probabilities add detail for the learning algorithm. section 5.3.1

Figure 5.8 The full version of RLHF. The dashed lines are text and require RL to update the parameters. The original LLM is the base model without any alterations, while the LLM to fine-tune starts as the base model but is altered to improve the quality of its outputs. The similarity and quality reward components are provided with word probabilities to improve calculation. RL adjusts the parameters by combining the quality and similarity scores.

Use the full pipeline diagram to compare the original base model with the model being fine-tuned and see how quality and similarity rewards jointly update parameters. section 5.3.3

RLHF is not a general solution to every failure. It is effective at avoiding known, specific problems, but it does not give the model new tools for novel problems. Optimizing a reward can also exploit weaknesses in that measurement, a form of Goodhart’s law. Testing remains incomplete because unknown risks cannot all be enumerated. section 5.3.3 section 9.1.1

RLHF commonly follows supervised fine-tuning rather than replacing it. It generally needs more examples—often tens of thousands—and can be applied sequentially with supervised fine-tuning. For code assistants, code data and supervised fine-tuning establish coding ability, while RLHF improves usefulness. section 5.2.3 section 6.1.1

How How Large Language Models Work covers it

How deep it goes

Explains agents, rewards, scoring, token probabilities, reward models, and stability, while deferring detailed RLHF mathematics.

Examples

Uses chatbot satisfaction, good-or-bad conversations, joke scoring, code assistants, and coding-platform data as teaching examples.

What it assumes

Assumes basic familiarity with LLMs and fine-tuning, while defining reinforcement-learning terms and reward mechanics from the ground up.

What you can do after

Evaluate when human-feedback fine-tuning can target abstract behavior and recognize reward hacking, novelty, and unknown-risk limitations.

How it connects

Connects reinforcement learning with human feedback to fine-tuning, supervised fine-tuning, reward models, Goodhart’s law, and adversarial machine learning.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
• 

  liveBookmanning.com
  LLMs in ProductionReinforcement learning from human feedback is a model-training approach that optimizes behavior using preference or evaluation signals supplied by people.
• 

  liveBookmanning.com
  Generative AI in ActionReinforcement learning from human feedback is a model-training technique that uses human preferences as feedback to optimize generated behavior.
• 

  liveBookmanning.com
  The AI Pocket BookA training method that uses human judgments as feedback to guide reinforcement learning and improve model outputs.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A training approach that uses human preference labels to optimize a language model toward responses humans prefer.
• 

  liveBookmanning.com
  Grokking AI ApplicationsReinforcement learning from human feedback is a training technique that optimizes model behavior using human evaluations of its responses.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionReinforcement learning with human feedback is a model-training technique that uses human evaluations to optimize a system's behavior.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionReinforcement learning from human feedback (RLHF) is a technique that uses human evaluations or preferences to estimate rewards and optimize model behavior.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionReinforcement learning with human feedback is a training approach that optimizes a model using human preference judgments converted into reward signals.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training setup that uses human preference feedback to build a reward signal for improving a model’s outputs.
• 

  liveBookmanning.com
  Transformers in ActionReinforcement learning from human feedback is a model-optimization technique that uses human preference feedback to define or improve a reward signal.
• 

  liveBookmanning.com
  First-Party Data ActivationReinforcement learning from human feedback is a model-training technique that uses human evaluations to shape a system’s behavior.
• 

  liveBookmanning.com
  Software Testing with Generative AIA model-alignment technique that uses human feedback to guide or refine a model's behavior.

Compare how 14 books explain reinforcement learning with human feedback →

Related concepts

• reward modeluses · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
• Proximal Policy Optimizationuses · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• reinforcement learninguses · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• Direct Preference Optimizationcontrasts with · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
• RLAIFalternative to · Reinforcement learning from AI feedback is a training or alignment technique that uses AI-generated evaluations or preferences as feedback to optimize a model or agent.
• ChatGPTused by · ChatGPT is OpenAI’s conversational AI service, which lets users interact with generative AI models and related tools through natural-language prompts.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.