1 Introduction
This chapter covers
- Why RLHF became important
- An intuition for how RLHF changes models
- An overview of RLHF and this book
Reinforcement learning from human feedback (RLHF) is a technique used to incorporate human information into AI systems. RLHF emerged primarily as a method for solving hard-to-specify problems. Such problems emerge constantly with systems designed to be used by humans directly, due to the often inexpressible nature of individuals’ preferences. This encompasses every domain of content and interaction with a digital system. The core idea to start the field of RLHF was, “Can we solve hard problems only with basic preference signals guiding the optimization process?” Its early applications were often in control problems and other traditional domains for reinforcement learning (RL), where the goal is to optimize a specific behavior to solve a task, and it became well known through the release of ChatGPT and the subsequent rapid development of large language models (LLMs) and other foundation models.