How deeply does Reinforcement Learning from Human Feedback cover three-stage rlhf recipe?

Defines all three stages and their data flow but leaves the reinforcement-learning optimization mechanics unspecified.

Three-stage rlhf recipe

As explained in Reinforcement Learning from Human Feedback

A three-stage RLHF recipe is a workflow consisting of instruction tuning, reward-model training, and reinforcement-learning optimization.

The three-stage RLHF recipe turns a text-completion model into a system optimized for human-preferred answers. It separates response-format learning, preference modeling, and reinforcement-learning optimization. section 1.3 section 1.7

The pipeline proceeds in three connected stages:

Instruction tuning: fine-tune the base model with next-token prediction on carefully crafted question-and-answer data, teaching it to follow prompts and produce an assistant-style response. section 1.3 section 1.7
Reward-model training: fine-tune a language model on preference relations among completions so it produces a scalar signal representing which answers people prefer. section 1.3
Reinforcement-learning optimization: use the instruction-tuned model to generate answers and optimize it against the reward model’s scores. section 1.3 section 1.7

From completion to assistant

A base model that merely completes text first sees high-quality question-and-answer examples. A reward model then learns from competing completions collected across prompts and labelers. Reinforcement learning uses its scalar judgments to improve the resulting answers. section 1.3

RLHF commonly changes response style, tone, and format while also improving benchmark performance. Over-optimization can produce excessive length or chattiness and harm capabilities in other domains. section 1.7

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

fine-tuninghas part · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
reinforcement learning with human feedbackimplements · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
reinforcement learninghas part · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
reward modelhas part · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.