4 Response surface methodology: Optimizing continuous parameters
This chapter covers
- Designing experiments to optimize continuous parameters
- Analyzing by building an interpolation model
- Analyzing by optimizing over the interpolation model
- Validating the optimal settings
A/B tests are straightforward and reliable. They are the “gold standard” of experiments, but there is a cost – i.e., the time, money, or risk involved in obtaining experimental results – to running them. Each of the subsequent chapters in this book presents a method that aims to reduce that cost. For example, multi-armed bandits adapt the experiment design continuously as new individual measurements are taken, and this reduces the time spent running the “wrong” version – A or B – of the system.
Response surface methodology (RSM) is specifically designed to optimize continuous parameters. RSM takes advantage of properties of continuous parameters to reduce experimentation cost compared to a more general method, like A/B testing.
The RSM procedure requires the experimenter to make decisions based, in part, on visualization of the business metric. These visualizations help make the procedure more transparent. This author believes that learning RSM first will lays a solid foundation for understanding more advanced methods that automate decisions and may seem more opaque to a user/engineer.