liveBook · Manning

Reinforcement learning

As explained in Fight Fraud with Machine Learning

Reinforcement learning is machine learning in which an agent learns parameter or action choices from reward signals associated with outcomes.

Reinforcement learning tunes a model toward outcomes that receive higher reward. In ChatGPT training, it follows unsupervised pretraining and supervised fine-tuning, using human judgments to improve generated answers. section 5.2.1

A reward-guided answer-generation loop works as follows:

  • Labelers rank, score, or choose between candidate answers to questions. section 5.2.1
  • A reward model learns to score a question-and-answer pair from those judgments. section 5.2.1
  • GPT generates an answer word by word; its answer and the question are scored by the reward model. section 5.2.1
  • The score produces a loss that is backpropagated to tune GPT parameters toward higher-reward answer sequences. section 5.2.1
Figure 5.13 Demonstration of Reinforcement Learning using Human Feedback (RLHF) to train ChatGPT

This workflow connects human answer preferences, reward-model scoring, generated sequences, and parameter updates. section 5.2.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains the reward-model and backpropagation loop, but defers algorithmic reinforcement-learning details.
Examples
Uses question-answer candidates ranked by labelers and compares training to a video game agent.
What you can do after
Evaluate/understand how human feedback can tune GPT parameters toward higher-scoring answers.
How it connects
Connects supervised learning, unsupervised learning, reward models, backpropagation, and autoregressive generation.

Also explained in

Compare how 28 books explain reinforcement learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.