preface

I became interested in AI during graduate school at UC Berkeley, where I arrived in the summer of 2017 and learned how deep learning was taking the computer science world by storm. I had no idea it was the same summer that the famous “Attention Is All You Need” paper, which introduced the Transformer, was released. Years later, after earning my Ph.D., I was working at the machine learning platform Hugging Face when ChatGPT captured the world’s attention.

This brings us to the topic of the book: reinforcement learning from human feedback (RLHF). RLHF burst onto the scene following the release of ChatGPT, serving as the crucial added technique that transformed GPT-3.5 into the ChatGPT we fell in love with. Over the last few years, I’ve been doing open research building models like ChatGPT, and I’m consistently shocked by how little of the information about how to do this is public—even basic definitions for common training algorithms. The goal of this book is to be the canonical reference for RLHF as it matures into an established area of research.