A/B tests are straightforward and reliable. They are the “gold standard” of experiments, but there is a cost—that is, the time, money, or risk involved in obtaining experimental results—to running them. Each of chapters 3-6 presents a method that aims to reduce that cost. For example, multi-armed bandits (MAB) adapt the experiment design continuously as new individual measurements are taken, and this reduces the time spent running the inferior version—A or B—of the system.
Response surface methodology (RSM) is specifically designed to optimize continuous parameters. RSM takes advantage of properties of continuous parameters to reduce experimentation cost compared to a more general method, like A/B testing. Both A/B testing and RSM help the engineer optimize a system by experimenting on it, but RSM has a narrower scope than A/B testing.
The RSM procedure requires the experimenter to make decisions based, in part, on visualization of the business metric. These visualizations help make the procedure more transparent. We believe that learning RSM will lay a solid foundation for understanding Bayesian optimization, which incorporates ideas from RSM.