A confidence interval is a range around an aggregate measurement that expresses plausible values for the underlying quantity. When the measurement is approximately normally distributed with standard error SE, the 95% interval is commonly approximated as measurement ± 2 × SE. section 4.1.5
In response surface optimization, compare an interpolated estimate with the validation measurement’s interval. If the estimate falls inside the bounds, the estimate and measurement agree closely enough to validate the result; otherwise, investigate the model or redesign the experiment. section 4.1.5 section 4.2.2
Validating an estimated optimum
A validation experiment measures markout profit as 0.37 with standard error 0.026. Its approximate 95% confidence interval is [0.32, 0.43]. An interpolated maximum of 0.41 lies inside that interval, so the result is validated. In a one-parameter example, a measurement of 0.158 with standard error about 0.009 produces bounds near [0.140, 0.176], containing the estimate 0.173. section 4.2.2 section 4.1.5
Use confidence intervals as an operational validation check:
- Measure the business metric at the proposed optimum and compute its standard error. section 4.1.5
- Calculate approximate 95% bounds as aggregate measurement − 2 × SE and aggregate measurement + 2 × SE. section 4.2.2
- Accept the optimum when its estimated metric lies within the bounds; otherwise shrink the modeled range or return to experiment design. section 4.1.5 section 4.2.2
Figure 4.12 Validate stage. In this final stage, you’ll run the production prop trading system at the optimal value of threshold that you estimated in the analyze stage. If the measured markout profit at that threshold is close enough (see the next section for discussion) to the estimated markout_profit(threshold), then you may accept the optimal threshold as the result of the RSM process.
The validation-stage diagram places confidence bounds in the final production check: compare the measured metric with the modeled estimate before accepting the parameter setting. section 4.1.5
