foreword

There is a gap in AI that almost nobody talks about openly: between what the leading, closed labs know and what is written down in public. A technique becomes standard practice at a few for-profit organizations. It gets a paragraph in the ``Methods'' section of a paper. The practitioners who implemented it move on. A year later, a team somewhere else spends four months reproducing what could have been explained in an afternoon. Nobody wrote about it. It wasn't anyone's job.

Nathan Lambert made it his role to communicate and document this science, and this book is the summary of all the attempts he has made to do so regarding large language models. Nathan joined Hugging Face in May 2022. We had just finished BLOOM---the biggest fully open language model run to date---with a coalition of hundreds of researchers across dozens of institutions; the science team was new, and nobody quite knew what came next. Nathan came to Paris that summer for the first science off-site. Over the year and a half that followed, across the eight- or nine-hour time zone gap between Europe and the U.S. West Coast, we had a lot of conversations about running, about research, about where the field was heading, and about what it meant to do good science when everything was moving faster than anyone could write it down.