Response surface methodology

Response surface methodology (RSM) optimizes a business metric over controllable continuous parameters while reducing the number of costly experimental settings. Rather than directly measuring every candidate setting, it measures a limited design, fits a surrogate function that interpolates between observations, and searches that model for a promising optimum. The result is an estimate, not a production fact, so validation is required. section 4.0 section 4.3

Figure 4.1 Overview of response surface methodology (RSM). RSM is an iterative process. First, in the design stage, we choose parameter values to measure. Next, we run the experiment to measure the business metric at those parameters. In the analyze stage, we use RSM techniques—interpolation and optimization—to estimate which parameter value is optimal. If the analysis fails, we return to the design stage. Otherwise, we validate the estimated optimal parameter value by measuring its business metric. If the measurement does not match the estimate, we return to the design stage. Otherwise, RSM has produced an optimal parameter value.

RSM is an iterative loop: design parameter settings, measure the business metric, interpolate and optimize a surrogate model, then validate or redesign. section 4.1

Use RSM as a disciplined experiment cycle:

• Constrain the parameter range with domain knowledge and choose settings that support later interpolation. Continuous parameters are ordered and have interior values, making estimates between measurements meaningful. section 4.1.1
• Collect replicated, randomized production measurements at each chosen setting. Favor fewer aggregate parameter settings, while taking enough individual measurements to make their standard errors small. section 4.1.2
• Fit a surrogate function to estimate the metric at unmeasured settings, then optimize the surrogate rather than repeatedly running costly experiments. section 4.1 section 4.1.4
• Measure the selected setting and accept it only when the validation result agrees sufficiently with the estimate; otherwise redesign around the new information. section 4.1 section 4.1.5

A useful surrogate must reflect a locally plausible response shape. In one dimension, at least three measured values are needed for interpolation to place a maximum between measured settings; two points only yield a line whose maximum is at an endpoint. RSM therefore trades more sophisticated analysis for fewer aggregate experimental configurations than broad comparison methods. section 4.1.1 section 4.1.2

For two or more parameters, preserve the same stages but expand the experiment design and model. A central composite design supplies center, edge, and corner settings for threshold and order size; a quadratic regression adds linear, squared, and cross terms to model behavior across the surface. The approach is best suited to three or fewer parameters, and unexpected outlying measurements can indicate that the fitted region is not near an optimum, requiring another iteration. section 4.2 section 4.2.1 section 4.2.2

Practical safeguards:

• Treat domain constraints as part of the design, not as an afterthought; invalid settings can make a system fail or create unacceptable risk. section 4.1.1
• Do not deploy solely from the surrogate optimum: it is an educated guess derived from a model rather than direct production measurement. section 4.1.4
• Use a contemporaneous A/B validation when measurements may drift over time, comparing the current system with the RSM-tuned version. section 4.1.5
• Restart with a better design when analysis finds no optimum, validation disagrees, or outliers violate the near-optimum modeling assumption. section 4.1 section 4.2.2