welcome

This book is written for anyone interested in taking an AI model from its origin state within the confines of a research paper or developer environment to safely introduce it in a production user facing setting by understanding its behavior, performance and impact using multiple distinct strategies.

As I write this, I’m probably thinking about an A/B test that went awry or maybe, more likely, dreaming of how to improve the offline evaluation strategy at my own job. And as you read this, I hope you find the need for improving how you evaluate the integrity, utility and overall behavior of an AI model just as important as the model itself.

Evaluations are what keep an AI model honest. A model might look promising in a research paper or even perform well on a benchmark dataset, but those snapshots don’t tell the whole story. Without thorough evaluation, you don’t know how the model will behave with messy, real-world data, edge cases, or under production constraints like latency.

The book is split into four parts. Every chapter includes a Jupyter Notebook (in github) that walks through example formulas to show the evaluation strategy in practice.